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SYSTEM FOR REQUESTING MISSING FIG. 16 is a block diagram of the flow aggregation 

NETWORK ACCOUNTING RECORDS IF processor (FAP). 

THERE IS A BREAK IN SEQUENCE FIG. 17 is a flow diagram of the flow aggregation process 

NUMBERS WHILE THE RECORDS ARE performed by the FAP of FIG 16 

TRANSMITTING FROM A SOURCE DEVICE s FIGS lg _ 20 are examples of ^ FAp enhancement and 

aggregation portions of the flow aggregation process shown 

BACKGROUND "> FIG - ir 

... , , , . f . FIG. 21 is a hie rarchical representation of an aggregation 

Inis invention relates to information management for j* * , , . „ P S t . - t 

. , t , ™, . t . B in adjustment scheme for adjusting the aggregation activity at 

Internet protocol (IP) packet transmission. 10 *u i i *• *u a ? JlL ! t 

r x 7 r the levels of the flow aggregation processor and the data 

Data collection systems are used to collect information collectors 

from network traffic flow on a network. These data collec- . , c ~ . - 

J j4 . . c ^ t FIG. 22 is an example of a configuration file update for 

tion systems are designed to capture one type of network *• / v \ j- * , u^uow W i 

. «: r . Vj j i * l j aggregation (policy) adjustment, 

tratnc from one source type and deliver the data to one . y 

application type such as a billing application. 15 FIG * 23 k a flow charl of 80 information management 

process. 

SUMMARY FIG. 24 is a representation of a network communications 

According to an aspect of the invention, a method for P ath between two end stations in a network, 

tracking network accounting records in an accounting pro- FIG. 25 is an illustration of an ICMP message encapsu- 

cess that collects and correlates information derived from 20 lated in an Internet Protocol (IP) packet and the formats of 

network data includes producing a network accounting the ICMP message and the IP packet, 

record that has an identifier that uniquely identifies the FIG. 26 is an illustration of the format of an ICMP error 

record within the accounting process with the identifier reporting message header and datagram prefix, 

including a sequence number that specifies a sequence FIG. 27 is a flow probe IP packet processing mechanism, 

number for network accounting records that originate from nG M is ^ payk)ad processing/protocol correlation of 

the source device and determining when there is a break m ^ lP packet processing mechanism of FIG. 26. 

the sequence numbers of network accounting records pro- CTr , e ~ nA - ftr> , . 4 . , 

duced from the device. The method also includes requesting . F IGS , 2 , 9A " 2 ^ , are f 1 ^™* de P 1C * a P rotoco1 

missing network accounting records when there is a break in ^ ^ 

the sequence. 30 FIG ' 30 15 a diagram depicting a process to capture quality 

of service 

One or more of the following advantage may be provided 

by one or more aspects of the invention. FIG " 31 ^ a diagram^of a service management process. 

The records produced in the accounting system have a FIG ; 3 ? * a showin g ™ architecture of a service 

sequence number that allows components that are in the next 35 P r0ViS10mn g application, 

level to detect if there are missing records in a collection of DETAILED DESCRIPTION 

records and can be used to give a sense of how often records Architecture 

are produced in a given time period, with this information Referring now to FIG. 1, an exemplary arrangement 10 

being part of every record, an accounting process can for collecting information from a network is shown. The 

determine a sense of the functional capabilities of the 40 net work includes various network devices 12. The network 

intermediate components and detect some aspects of the devices 12 ca n be disparate, i.e, different equipment types, 

communication channel between components. operating under different protocols and formats. The net- 

BRIEF DESCRIPTION OF THE DRAWINGS work devices 12 are coupled to an accounting process 14 via 

an equipment interface 16. 

FIG. 1 is a block diagram of a server running an account- 45 The accounting process 14 includes a flow data collection 

ing application monitoring a network. i ayer 18 mat ^ client processes with the equipment 

FIG. 2 is an architectural block diagram of the accounting interfaces on or close to the network devices 12. Individual 

application used in FIG. 1. and multiple data collectors (not referenced) can be disposed 

FIG. 3 is a block diagram of accounting support in an at points of presence (POP) in a network 11. The accounting 

access process used by an Internet/Intranet service provider 50 process 14 includes a flow aggregation and distribution 

or a large enterprise. process 17 that runs as a server process on a server 15. The 

FIG. 4 is a block diagram of accounting support in an accounting process 14 assembles the data into a format that 

access process used by an Internet/Intranet service provider can ^ e used bv billing or other user defined data consuming 

or a large enterprise using an Extranet switch. applications 20 that interface to the accounting process 14, 

FIG. 5 is graph depiction of a network including data 55 through a data consuming application interface 22. Thus, the 

collectors disposed in the network. accounting process 14 collects via the data collector layer 18 

t - a j * multiple and diverse types of data from the network 11, 

FIG. 6 is a now diagram showing a typical data flow , . . ' r . , A t . , j 

flmMCC „•„ „ M „ & /r normalizes the data into a consistent accounting record, and 

process using an accounting process. ._. - ■ . 

- . . provides open interfaces to one or more applications, such as 

FIG. 7 is a diagram show exemplary network accounting 60 billing via thc application interface 22. 

records. The network devices 12, e.g., switches, routers, remote 

FIGS. 8A-8B, 9A-9B, 10, 11A-11E, 12 and 13A-13B, access concentrators, and so forth can produce data of 

are schematic views of data structures used in network various types and formats which are all handled in the 

accounting records. accounting process 14. Examples of the network devices 12 

FIG. 14 is a block diagram of a flow data collector system. 65 include a router or switch 12a, cable or telephone modems 

FIG. 15 is a flow diagram of the flow data collection 12b, a flow probe 12c, a remote access concentrator 12d an 

process of the flow data collector of FIG. 14. Extranet switch 12e, a directory naming service (DNS) 
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server 12/, a RADIUS server I2g and web server \2h. One can support a so-called "push model" in which a connected 

particular source of data, the flow probe 12c will be device initiates a transmission of data to the accounting 

described below in conjunction with FIGS. 24-28. The process 14. The data collectors S2a-S2g also can support a 

network devices 12 can include a "Remote Authentication "p U u model" in which the accounting process 14 initiates a 

Dial -In User Service" (RADIUS) server 12g that produces 5 connection to an equipment interface for the purpose of 

RADIUS accounting records using an existing RADIUS obtaining data. In addition, the data collectors S2a-S2g can 

accounting process (not shown). The accounting process 14 support an "event driven model" in which an event that 

can interface to the existing RADIUS accounting process 0CCUR ^ eithcr ^ cqu i pmcnt interface layer 16 or in the 

and can use existing RADIUS records without modifying accounting process 14 initiates a transfer based on some 

the existing RADIUS accounting environment. RADIUS is 1Q of cd{c ^ bcm m{ b ^ ^ , lfi 0f 

a well-accepted standard in the industry and is used across accmmti ^ 14 ^in which the event occurred. 

nSr™ f T TZ 1 *?Tn^ ™* data collectors S2a-S2g are distributed throughout 
DSL, VOIP, etc.), with the most prominent being dial-in * 0 
access. So, by supporting RADIUS records the accounting 1116 n ^ wo *- Ibe data collectors S2a-S2g are placed close to 
process 14 provides the ability to fit into an existing network or * e netwo * *vice *e collector is assigned to. 
environment without modification. 15 ^ data collector can be in-line or out-of-hne 
Toe accounting process 14 enables users such as an relative to the device monitored. The data collectors 
Enterprise or an Internet Service Provider to maintain an S2a-S2g can be anywhere. The data collectors 52a-S2g can 
existing accounting configuration. Information sources can be completely uncoupled from the devices except for corn- 
include network traffic flow, RADIUS accounting data, munication paths. As new network devices 12 are added to 
RMON/RMON2 data, SNMP-based data, and other sources 20 the accounting support arrangement 10, new data collectors 
of network usage data. The accounting process 14 collects are also deployed. 

data via the data collector layer 16 from multiple disparate The accounting process 14 also includes a flow aggrega- 

sources and produces new type of composite records. These lion process 60 that is part of the aggregation and distribu- 

new composite records results is new information which tion process 17 (mentioned above). The flow aggregation 

provides a source for network accounting, billing, 25 process 60 is a central collection point for all network 

management, capacity planning, and so forth. accounting records (NAR's) produced from various data 

The accounting process 14, as will be described in FIG. 2, collectors 52a-S2g in the data collection layer 18. The flow 

has a core process that can handle data records from each of aggregation process 60 receives NAR's from various data 

the equipment types above, as well as other equipment collectors S2a-52g and aggregates, i.e., summarizes related 

types, and can provide data to each of the plurality of 30 information from the received NARs across the accounting 

user-defined data consuming applications. This accounting support arrangement 10. The aggregation layer 60 produces 

process 14 provides flexibility in choosing data consuming Summary NAR's i.e., enhanced and unique network 

applications that use accounting data. Such applications can accounting records. That is, the flow aggregation process 

include billing, enterprise charge -back or cost allocations, aggregates the records across the network devices; whereas, 

capacity planning, trending, application monitoring, user 35 individual data collectors S2a-52g can aggregate accounting 

profiling and so forth. records from individual data sources. Aggregation will be 

Accounting Architecture described below in FIGS. 14-23. 

Referring now to FIG. 2, the equipment interface layer 16 The accounting architecture also includes a data distribu- 
of the accounting process 14 includes various equipment tor layer 70 (part of the aggregation and distribution process 
interfaces 42a-42t which are, respectively, an interface 42a 40 17). The data distribution layer 70 provides a flexible data 
for the router/switch 12a, an interface 42b for the cable/ distribution mediation between the flow aggregation process 
modem head end 12b, and an interface 42c for the flow 60 and a user-defined application, via an application inter- 
probe 12c. The equipment interface layer 16 also includes face layer 22. Data distributor layer 70 presents information 
additional interfaces such as an interface 12d for a remote to the application interface layer 22, with a pre-defined 
access concentrator 12*/, an interface 12e for an Extranet 45 format, protocol and schedule that is determined by require - 
switch 12e, an interface 42/ for a DNS server 12/, and an ments of a user application. The data distributor layer 70 can 
interface 42g for a RADIUS server 12g. The equipment support push, pull and event driven data distribution models, 
interface can have additional interfaces that can be specified, The application interface layer 22, is comprised of indi- 
as new equipment is added. The interfaces 42a-42g can be vidual application interfaces S2a-82g that are provided by 
developed by an interface toolkit 44. The interface toolkit 44 50 the toolkit 44. The toolkit 44 as with the network device 
permits a user to construct a new equipment interface type interfaces 42a-42g can be used to produce additional appli- 
to couple the accounting process 14 to a new equipment cation interfaces 82. 
source type. Exemplary Configurations 

The accounting process 14 also includes a data collector Referring now to FIG. 3, the accounting process 14 can, 
layer 18. The data collector layer 18 is a distributed layer 55 in general, support any configuration. Exemplary configu- 
comprised of individual data collectors, e.g., S2a-$2g, The rations used by an Internet service provider 100, an Enter- 
data collector layer 18 collects data in the form of raw prise A that host its own remote access 110, and an Enter- 
accounting information specific to the device type. The data prise B that uses the Internet service provider 120, are 
collector collects data via the aforementioned equipment shown. 

interfaces 42a-42g. The data collectors S2a-S2g collect the 60 As shown in FIG. 3, for the Internet service provider, data 

data and convert data into normalized records herein collectors 52a-52d are distributed at specific Points of 

referred to as Network Accounting Records (NARs). Each of Presence (POP), such as remote access concentrators 102 

the data collectors S2a-52g, as appropriate, forwards net- managed by the Internet service provider. The remote access 

work accounting records (NARs) to a flow aggregation concentrators allow, a mobile user 106 or an Internet user 

process 60. 65 107 with remote access to access an enterprise over the 

The data collectors 52a-52g support several different Internet, via the Internet service provider. In this example 

collection models. For example, the data collectors S2a-S2g the Internet service provider arrangement 100 and the large 
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Enterprise arrangements HO and 120 include servers 13, 13', configuration, the Extranet switch 122 can be owned and 

and 13" that run accounting processes 14, 14' and 14". The operated by an Internet service provider as shown with 

accounting processes 14, 14* and 14" each independently enterprise A, or it could, alternatively, be owned and oper- 

manage and collect information regarding network traffic ated by an enterprise, as shown with enterprise B. Users 

usage. S would access the corporate network of either enterprise A or 

The Internet service provider arrangement 100 includes enterprise B t via the Internet service provider with various 

the accounting server 13 that runs the accounting process 14. types of tunneling protocols such as L2TP, L2F, PPTP or 

The accounting process 14 includes a flow data collector IPSec, and so forth. The accounting server 13 located at the 

layer 18 that gathers data from the service provider network service provider and also accounting servers 13', 13" within 

100. The flow data collector layer 18 includes distributed, 10 enterprise A and enterprise B allow each the Internet service 

individual flow data collectors S2a-S2d, The distributed, provider and each of enterprises A and B to run accounting 

flow data collectors S2a~S2d collect transaction specific process 14', 14" on the servers 13', 13" to monitor and collect 

details about a user's connection type and actual network network data, 

usage. These data are converted into the NARs in the Transaction Row Model 

distributed, flow data collectors $2a-S2d> as mentioned 15 Referring now to FIG. 5, a graph 140 depiction of a very 

above. The NARs are aggregated over the entire system by large scale network includes a device "A" 142 communi- 

the flow aggregation layer 60 (FIG. 2). eating with a device "B" 144. The graph 140 includes nodes 

Data is made available to the Internet service provider via (not all numbered) that can represent routers, switches, flow 

the data distribution layer (FIG. 2) so that the Internet probes, etc. that have interfaces (not shown) which maintain 

service provider can analyze data in order to differentiate 20 statistics on information passed through the interfaces. For 

service offerings to different users. The Internet service example, a switch may have a number of Ethernet ports and 

provider can evaluate and use such detailed accounting data a host could be connected to one of the ports and in 

collected from various sources to migrate from a single flat communication with one of the interfaces to transfer infor- 

rate fee business model to more flexible charging. For mation over the network. The interface would have counters 

example, analysis of the data can enable the Internet service 25 that are used to track "packet's in, "packet's out", "bytes in, 

provider to develop new service options that can take into bytes out", and so forth. 

consideration bandwidth usage, time of day, application In this case of the host connected to the port, or a router 

usage and so forth. In addition, Internet service providers or some other device being connected to the port, there is no 

can offer discounts for web hits that may exist in an Internet other connection that the host, router or other device is 

service provider cache, thereby minimizing the need to look 30 aware of other than the entire network. This is an example 

up an address for a requested site on the Internet and can of a "connectless oriented" protocol. A data collector 52 can 

provide free E-mail usage while charging for other types of be disposed.in the network in a path between the entities "A" 

applications such as file transfer protocol and web traffic. and "B", such that the data collector 52 monitors some of the 

The data can also be used by other applications such as packets that comprise a flow between "A" and "B." As a 

network planning, security, auditing, simulation, flow pro- 35 single point monitor, the data collector has no concept that 

filing capacity planning and network design and so forth. there are two ends communicating. The data collector 52 

Thus, the Internet service provider can independently moni- identifies these entities "A" and "B" in various NARs 

tor and evaluate network traffic caused by remote employees produced by the data collector 52. At later stage in the 

and mobile users, for example. processing, either in the data collector 52 or elsewhere in the 

Similarly, other instances 14', 14" of the accounting 40 accounting process 14 the NARs are correlated so that the 

process can be used by enterprises, as also shown in FIG. 3. NARs or some aggregated NAR produced by the data 

For example, an enterprise may host its own remote access, collector 52 or the rest of the accounting process 14 can be 

as shown for Enterprise A and would include a server 13' associated with the accountable entities "A" and "B" to thus 

running an accounting process 14'. An enterprise could use identify a connection between entities "A" and The data 

the Internet service provider as shown for Enterprise B, and 45 collectors S2a-S2g (FIG. 2) develop a description of the 

still have a server 13" running an accounting process 14". connection. For a router, normally an address of the object 

The accounting process 14', 14" includes an associated data that is connected to that interface might serve as an address, 

collector that is coupled to enterprise A and enterprise B A switch has an IP address that can be used as the destina- 

local area networks or other network arrangement. In this tion. The actual device that is connected to the switch or 

model, the enterprises use data from the accounting process 50 router can be used as an accountable object. Although the 

14', 14" for enterprise charge-back functions such as billing traffic is not destined for the router, the data collector can use 

departments for Internet usage within the enterprise and so those identifiers as keys to the endpoints "A", "B." 

forth. In many cases, the protocol can explicitly determine 

Different instances of the accounting process are used by connections. For example, the TCP/IP protocol is explicitly 

both the Internet service provider and enterprise A and 55 a "connection oriented" protocol used in the Internet. When 

Enterprise B sites. The instances 14, 14", 14" of the account- the data collector 52 needs to determine a connection, the 

ing process are independent they do not need to exchange data collector 52 can exploit the "connection oriented" 

accounting data. Rather, they exist as separate, independent nature of certain types of protocols such as the TCP/IP 

accounting domains. protocol. When the data collector 52 tracks a TCP/IP 

Referring now to FIG. 4, a similar access configuration 60 connection, the data collector 52 can determine exactly that 

100', as the configuration 100 (FIG. 3) can be used with an A and B are connected, when the connection starts, stops, 

Extranet switch 122. Extranet access allows remote users to and updates. With other protocols such as a "connectionless" 

dial into an Internet service provider (ISP) and reach a protocol, and even in some complex environments such as 

corporate or branch office via an ISP. The Extranet switch a virtual private network or a proxy server, the data collector 

allows Internet users access to corporate databases, mail 65 52 does not necessarily know the real endpoints. The data 

servers and file servers, for example. It is an extension of the collector 52 only knows that some entity is talking to some 

Internet in combination with a corporate Intranet. In this other entity. 
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Thus, the data collector 52 is a single point monitor, that 
monitors traffic at one point in the network and converts the 
traffic into a "pipe oriented" or "flow oriented" accounting 
information. The data collector 52 identifies a source and a 
destination of the traffic. That is, the data collector develops 
a "connection oriented tracking." By distributing data col- 
lectors S2a-S2g (FIG. 2) through out the network the 
network can be modeled as pipes having two endpoints. A 
data collector can be disposed in a partial pipe. The data 
collector 52 determines that one end of the pipe refers to "A" 
and the end of the pipe refers to "B." The data collector 52 
can be disposed anywhere along the network. 

The graph 140 represents the network as a directed graph, 
including partial segments. The endpoints of those partial 
segments can act as proxy entities to the actual accountable 
objects. Once independent accounting records that relate to 
these two entities A and B are aggregated in the accounting 
process 14, the accounting process 14 can identify that A and 
B are connected and have particular metrics. 

Some equipment have a half pipe model that generate 
independent accounting records for each half pipe. The data 
collectors can assemble full pipe information from half pipe 
information. The accounting process could be coupled to 
equipment that gives a half pipe model for A communicating 
with B and a separate one for B communicating with A. The 
data collectors 52a-52g combine information from these 
two half pipes into a bidirectional flow. 

Referring now to FIG. 6, an example of data flow 130 
through the accounting process 14 is shown. In this example, 
data flow is initiated by a user 131 making a call to a remote 
access concentrator (RAC) 132. Upon receiving the call, the 
RAC 132 authenticates the user against a secure access 
controller 134. After verification, the RAC 132 connects the 
user to the network 135 and sends a RADIUS Start record 
(not shown) to the accounting process 14. The accounting 
process 14 generates a RADIUS Start NAR 137a and stores 
the RADIUS start NAR in a database 62. At that point, the 
remote user may check e-mail, look at a web server and 
transfer a file. For each transaction, the accounting process 
14 captures !he IP traffic, generating & e-mail, http, and ftp 
network accounting records \37b-X37d, respectively. These 
are stored in the database 62. Upon completion of these 
transactions the user would log out of the network, at which 
time the RAC would send the accounting process 14 a 
RADIUS Stop record. The accounting process 14 generates 
a RADIUS Stop NAR 137e and stores the RADIUS stop 
NAR in the database 62. All of these records reflecting the 
user's transactions could be viewed and reported in flexible 
ways dependent on the needs of an end-user application. 
Network Accounting Records (NARs) 

The data collector 52 translates collected information into 
network accounting records (NARs). A NAR includes a 
header, an accounting entity identifier and metrics. The 
network accounting record (NAR) is a normalized data 
record that contains information about a user's network 
usage activity. 

Referring now to FIG. 7 a base level "activity" NAR that 
includes source, destination, protocol, source port, destina- 
tion port, byte and packet counts, etc. The base level activity 
NAR can be combined and aggregated in many different and 
flexible ways to support various accounting functions. The 
NAR is an activity record corresponding to some level of 
detail. Detail can vary based on the level of aggregation 
being applied, in accordance with the needs of the end-user/ 
application. 

FIG. 7 has at one level 152 a plurality of exclusively 
"Activity NARs" which could correspond to a very low 
level of detail, or could be the result of a prior aggregation 
providing a higher level view of the information. Thus, FIG. 
7 shows a collection 152 of exclusively activity NARs. From 
base level data, additional "views" of the NAR could be 
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produced, such as a set of "Summary NARs" 154, or another 
set of Activity NARs 156 which could be a result of further 
aggregation of the base level information, or lastly a com- 
bination of a set of Summary NARs and Activity NARs 158, 
The summary NAR is produced by the central aggregation 
layer 60 and can include user identifying information, pro- 
tocol information, connection time information, and data 
information, and so forth. 

The specifics of what can be combined and aggregated 
will described below. Thus, the accounting process 14 use of 
NARs provides the ability to combine and aggregate base 
level activity information in a flexible way to meet the 
specific needs of the end-user/application. 

TABLE 1 below corresponds to the fields that can be 
captured in a NAR. This is essentially the activity NAR. The 
NAR contains these fields, which can then be combined and 
used to form other activity NARs or summary NARs. 



20 



Column Name Description 



OSA_SOURCE_TYPE 



25 



OSA_SOURCE_SERIAL_J*UM 



35 



START_TTME_SEC 



START_TTME_USEC 



30 SEQU ENCE_NUM BER 



USER_NAME 
EVENT 

SESSION_ID 

SESSION_TIME 
SESSION__TIME_USEC 

SRC_ADDR 
DST_ADDR 
PROTO 

SRC_PORT 
DST_PORT 
SRC_OCTETS 



45 



DST_OCTETS 



50 



SRC_PKTS 



DST_PKTS 



55 



SRC TOS 



DST_TOS 



60 



SRC_TTL 



DST_TTL 



List all of the possible data sources 
from which data can be collected. 
Reference to 

OSA_SOURCE_TYPE TABLE. 
Number which uniquely identifies 
an OSA FDC. 

Indicates the date and time at 
which a record was recorded. 
Microseconds component of 
START_TI M E_S EC. 
Sequence number assigned by the 
source of the NAR to uniquely 
identify the record and ensure 
-integrity. 

The user associated with the record. 

Event type of the record (i.e. 

Start, Stop, Update). 

Unique Accounting ID to make it 

easy to match start and stop records. 

Duration of the record in seconds. 

Microseconds component of the 

SESSI0N_TIME. 

Source address of the record. 

Destination address of the record 

Protocol of the record. Reference to 

OSA_PROTOCOL_TYPE table. 

Source port number. 

Destination port number. 

Number of bytes transmitted into 

the network by the source. For 

RADIUS is equivalent to 

Acct-Input-Octets . 

Number of bytes sent out of the 

network, to the source. For 

RADIUS is equivalent to 

Acct-Output-Octets. 

Number of packets transmitted into 

the network by the source. For 

RADIUS is equivalent to 

Acct-Input-Packets. 

Number of packets transmitted 

out of the network, to the source. 

For RADIUS is equivalent to 

Acct-Output- Packets. 

The Type of Service coding marked 

by the source. 

The Type of Service coding 

marked by the destination. 

The Time To Live field set 

by the source and modified by the 

network until the Nortel flow 

probe recorded it. 

The Time To Live field set by 

the destination and modified by the 

network until the Nortel flow probe 

recorded it 



03/04/2004, EAST Version: 1.4.1 



US 6,625,1 

9 

-continued 

Column Name Description 

OSA_CAUSE A number that indicates the reason $ 

the accounting record was generated. 

OSA_STATUS A value used to indicate the status 

of an accounting record when 
it was created. 

ACCT_DELAY_TiME Indicates how many seconds the 

client has been trying to send 10 
this record 

ACCT_AUTHENTIC Indicates how the user was 

authenticated. 

ACCT_TERMINATE_CAUSE indicates how the session 
was terminated 

ACCT_MULTI_SESSION_ID Unique Accounting ID to make 55 
it easy to link 

ACCT_UNK_CO UNT Indicates the count of links which 

are known to have been in a 
given mululink session at the time 
the accounting record is generated. 



The summary NAR and activity NAR have a one -to-many 
relationship. That is, while there can be a single summary 
NAR for a particular user over a particular call that will 
contain information about the sum of usage of network 
resources over the duration of the call, there can be many is 
activity NARs. The activity NARs capture details about the 
actual activity and applications being used during the call. 
The summary NAR, therefore, depicts the total expense of 
the transaction or a set of transactions on a network, 
whereas, the activity NARs depict expenses of a transaction 30 
at any point in time. The summary NAR is generated in the 
flow aggregation.process-60, as will be described below. In _ _ 
essence, the summary NAR is generated from individual 
activity NARs correlated in the data collectors 52a-52g, as 
will be described below. 35 

A NAR is a member of a generic data message set that is 
used to transport data, such as network accounting data, 
through the accounting process 14. These system data mes- 
sages include "Status Event", "Maintenance Event", "Trace 
Event", "Network Accounting Record". Accounting process 40 
14 messages share a common MSG_HDR structure that is 
used to discriminate between the four types of accounting 
process 14 messages. The Message Header (MSG_HDR) 
includes Message Type, an Message Event and Cause, and 
Message Length. 45 
Network Accounting Record Data Structures 

As will be described below, the NAR is unique within the 
accounting process 14. The NAR has a NAR_ID that 
specifies an accounting process component ID. The compo- 
nent ID specifies the data collector assigned to a particular 50 
network device that produced the NAR. The component ID 
e.g., NAR_SRC_JD 203fl (FIG. 8B below) is allocated at 
the time that the component is produced. The NAR_ID also 
includes a time stamp at the second and microsecond level 
so that the accounting process 14 can discriminate between 55 
multiple NARs generated by a particular component. A 
sequence number, e.g., 32 bits is also used to differentiate 
NARs from the same accounting process component that 
may have the same time stamp. The sequence number e.g., 
NAR__SEQ_NUM 203c (FIG. 8B) is preferably a mono- 60 
tonically increasing number starting from, e.g., 1. As long as 
the component is functioning, and producing NARs, the 
component provides a new sequence number to a new NAR. 

Referring now to FIGS. 8A-8C, a Network Accounting 
Record (NAR) data structure 200 is shown. 65 

As shown in FIG. 8A, the NAR data structure 200 
includes two basic accounting objects, a Network Account- 
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ing Record Identifier 202, and optionally one or as shown a 
plurality of Network Accounting Record Attributes 
204a-204/i, generally denoted as 204. The Network 
Accounting Record Identifier 202 has a set of object iden- 
tifiers that uniquely identifies the network accounting record 
within the accounting process 14. 

The Network Accounting Record Identifier 202 acts as a 
database key value that makes the NAR 200 unique within 
the entire accounting process 14. The Network Accounting 
Record Identifier 202 allows the NARs to be handled and 
managed using database functions such as database integrity 
analysis and reliability analysis. The Network Accounting 
Record Identifier 202 also gives the accounting process 14 
the ability lo track the source of NARs and to build mecha- 
nisms such that the accounting process 14 can maintain 
identity of the origination of NARs throughout the system 
10. 

The plurality of Network Accounting Record Attributes 
204a-204/i provide metrics for the NAR 200. The Network 
Accounting Record Attributes 204a-204/i capture specific 
information contained in data from network devices. Dif- 
ferentiating between the entity identifier 202 and the metric 
204 allows the accounting process 14 to perform logical and 
arithmetical operations on metrics 204 while leaving the 
accounting identifier intact 202. The accounting identifier 
202 can be enhanced unlike the metrics. 

The data collectors 52a-52g (FIG. 2) are oriented around 
the process of filling in the NAR. The metrics are left 
untouched by the data collector and are passed transparently 
into the accounting process flow aggregation process 60. 
The data collectors 52a-S2g assign the accounting entity 
identifiers 202 tojhejnetrics e.g., a source and a destination 
identifier to the metric. In the example of a router link, the 
metrics that the router interface provides are in the form of 
"information in" and "information out" e.g., octets in, octets 
out, bytes in, bytes out datagrams in, datagrams out, faults 
in, faults out, and so forth. The data collectors S2a-S2g 
determine what "in" and "out" means and assigns the unique 
identifier that is unambiguous relative to the determined 
meaning of "in" and "out." Once a data collector 52 has 
established this convention, the convention is used through- 
out the system 10. 

Thus, the NAR Identifier 202 provides database con- 
structs to a NAR, whereas, the plurality of Network 
Accounting Record Attributes 204a-204w provide the actual 
metrics used for network activity reporting and network 
accounting. 

As shown in FIG. 8B, the Network Accounting Record 
Identifier 202 (NAR_JD) is a set of objects within the NAR 
that uniquely identifies the NAR throughout the accounting 
process 14. The NAR_ID 202 is designed to support a 
number of properties of a NAR including flexibility, 
accountability, reliability and uniqueness. In order to pro- 
vide these properties, the NAR_ID 202 is divided into 
objects designed to specifically provide these properties. 
Flexibility is supported through a NAR_HDR 203 section 
of the NAR^ID. Accountability is attained in the NAR 
through explicit identification of the source of the NAR by 
a component identification NAR_SRC_ID 203a. The 
source time is maintained in a NAR_SRC_TIME 2036. 
Reliability is supported, as described above, through the use 
of a NAR sequence number (NAR__SEQ_NUM) 203c, 
which is designed to enable traditional database integrity 
mechanisms. 

The NAR_ID 202 is used to provide uniqueness for each 
NAR. The responsibility for guaranteeing the uniqueness of 
each NAR is handled by every accounting process compo- 
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nent that has the ability to originate/source network account- 
ing records. This responsibility requires that each account- 
ing process component have the ability to unambiguously 
identify itself in each NAR that it produces. Thus, NAR type 
identifier, NAR_TYPE, is comprised of the source compo- 
nent identifier, NAR_SRC_ID, the NAR source time, 
NAR_SRC__TIME, and the NAR sequence number, 
NAR_SEQ_NUM. These three data objects act as a data- 
base key for a particular network activity record, ensuring 
the uniqueness of the NAR throughout the entire system. 

The NAR_SEQ_NUM can have several purposes. One 
way that the NAR_SEQ_NUM can be used is as a dis- 
criminator when two NARs are produced at the same time. 
A second way that the NAR_SEQ_NUM is used is as a 
monotonically increasing index to ensure database integrity. 
Because the NAR_ID is unique, it should be considered as 
an allocated value. A NAR_ID is allocated at NAR origi- 
nation. 

If a component creates or modifies the contents of an 
existing NAR, as for example when aggregating two NARs 
together, the component originates the NAR_ID. This pro- 
vides an opportunity for the accounting process 14 to have 
explicit internal integrity mechanisms that can account for 
any network accounting record that is processed by the 
accounting process 14. 

The NAR Source Identifier NAR_SRC_ID 203a 
includes a source type 207a and a Source Serial Number 
207b. The serial number 207b is an administratively allo- 
cated value e.g., 24-bits that uniquely identifies the NAR 
source type throughout the accounting process 14. The 
source serial number 2076 should be unique within the 
specific accounting domain. 

The (NAR_SEQ_JWM) 203c is a -monotonically 
increasing, e.g., unsigned 32-bit integer that acts as a 
sequence number for NARs that originate from a particular 
NAR source. Because the value of the NAR_SEQ__NUM 
can "wrap around", the combined 64-bit value NAR_SRC_ 
ID and NAR_SEO_NUM are unique only over a specified 
time period. 

Referring now to FIGS. 9A-9B, exemplary formats for 
Network Accounting Record Attributes 204 are shown. 
There are two variations on a single NAR_A1T KIBUTE 
format that can be used. As shown in FIG. 9A, a standard 
NAR ^ATTRIBUTE format 206a includes header fields 
NAR__ATTR type, NAR_ATTR Code, NAR_ATTR 
Qualifier, and NAR_ATTR Length and a "value field." In 
order to conserve the size of accounting information, when 
the size of the value of the N AR^AJTRIB UTE is a byte i.e., 
8-bits, as indicated in the NAR-ATTR Qualifier field, the 
format 206b of the NAR^ATTRIBUTE can be as shown in 
FIG. 9B, including fields NAR_ATTR type, NAR^ATTR 
Code and an 8-bit NAR_value field. 

Each supported object is assigned an NAR_ATTR Code. 
Through the NAR _ATTR Code, the accounting process 14 
can distinguish the semantics of a particular NAR 
ATTRIBUTE. Although NAR_ATTR Codes are specific to 
the NAR__ATTR Type, the NAR_ATTR Code assignments 
can be unique to aid in implementation. Values can be 
assigned to provide some explicit hierarchical structure. 
Each NAR _ATTR has an 8-bit NAR_ATTR Qualifier that 
provides typing information for the NAR_^A1TK. The 
NAR_ATTR Qualifier is used because some supported 
objects can be represented using several different types. 
Counters, for instance can be 32-bit as well as 64-bit, in the 
case of aggregated objects. Network identifiers may use 
numeric indexes, or strings as labels. The NAR^AiTK field 
specifies the length of the NAR attribute including the 
NAR _ATTR header. 
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There are five types of Network Accounting Record 
Attributes that are supported in the NAR. The five attributes 
are Accounting Time Interval (ACCT_TIME) (FIG. 10); 
Accounting Entity Identifier (ACCT_ENTITY_1D), 

5 (FIGS. 11A-11E); Accountable Entity Descriptor (ACCT_ 
ENTITY_Desc); Network Activity Metrics (NET_ 
METRICS)(FIG. 12); and two Transparent Attributes 
(TRANS^ATTRXFIGS. 13A-13B). As necessary, addi- 
tional NAR_ATTRIBUTES can be supported. For example, 

10 a NAR_ATTRIBUTE type could also include Security 
Attributes for accounting data to protect against unautho- 
rized introduction or modification of accounting informa- 
tion. 

Referring now to FIG. 10, an Accounting Time Interval 

15 record includes a value "seconds" and a value "micro 
second". The values of "seconds" and "micro seconds" 
together represent a time stamp of network activity for the 
NAR, as discussed above. When derived from an absolute 
time value that represents the end of the accounting time 

20 interval, the Accounting Time Interval is the duration, as 
calculated using the Accounting Time Interval as the starting 
time value. All Network Accounting Records can have an 
Accounting Time Interval attribute. 

Referring now to FIGS. 11A-11E, Accountable Entity 

25 Identifier data structures are shown. The Accountable Entity 
Identifiers are a collection of entity description attributes 
that together identify an accountable entity in the accounting 
process 14. The accounting entity identification mechanism 
facilitates flexible NAR aggregation properties of the 

30 accounting process 14. The ACCT_ENTITY_ID is the 
description of an accounting object within the accounting 
process 14. There can be one or more ACCT_JENTITY_ 
IDs in a given NAR, but there must be at least one ACCT_ 
ENT1TY_ID in an Network Accounting Record. The actual 

35 accountable object is defined by the entire collection of 
ACCT_ENTITY_JDs that are included in the NAR. 

In transaction based accounting, a network accounting 
record will contain two ACCT__EN 11 lY__JDs, representing 
the source and the destination entities that are involved in the 

co network transaction. For traditional flow based accounting, 
these would normally be the two network addresses that are 
involved in the flow. Qualifiers are available in the ACCT_ 
ENTITY_ID objects to indicate which ID is the source and 
which is the destination of the network transaction. 

45 In direct support of flow based accounting data sources, 
the accounting process 14 supports a specific IP flow 
descriptor. This is the traditional IP 5-tuple flow description. 
The accounting process 14 could also support a 6-tuple flow 
descriptor that includes a type of service (TOS) indicator in 

50 the flow designator. This allows for Class of Service dis- 
tinction in the accounting model. 

For network activity data sources that do not have a 
transaction accounting model, there may only be a single 
ACCT_ENTITY_JD present in the accounting record. 

55 Qualifiers for the ACCT_ENTITY_I D are available to 
indicate if the single object is the source, destination, or 
both, for the accounting metrics that will be included. The 
types of entities include User Identifiers and Network Entity 
Identifiers. The network identifiers can include IP Address, 

60 Flow Description, and Network Object ID. Other types of 
accounting entities can be provided. 

The actual accountable entities for a specific network 
accounting record are specified in the complete set of 
ACCT_ENTITY_ID(s) that are present in the NAR. 

65 Operations that can be applied to NARs, specifically 
aggregation, can influence how ACCT_J:NTITY_JDs are 
used in NARs. Each accountable entity identifier that is 
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present adds refinement to the definition of what accountable 
entity the metrics actually apply to, whereas each ACCT_ 
ENTITY_DESC further refines the description of the 
accountable entity. 

Referring now to FIG. 11A, a NAR_USERNAME is a 
specific type of NAR_USER1D data structure. A system 
string type "Username" 222 can represents a real account- 
able user within the accounting process 14. The NAR_ 
USERNAME data structure 220 is used to transmit the 



are populated from transport and network layer of IP packets 
via flow probe. The NAR_FLO W_D ESC NAR_ATTR 
Qualifier provides for Role designation, indicating whether 
the object referenced is acting as a source, destination, both 
or undeterminable within the system. These bits are set when 
the Role can be determined without ambiguity. 

Therefore the Network Accounting Activity Records are 
fundamentally bindings between an accountable entity and a 
set of metrics that can be associated with that entity over a 



string. The semantics can be applied when the string "User- 10 specified period of time. The NARs provide flexibility in 
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name" 222 is supplied by RADIUS or from DCHP man- 
agement systems. The NAR_USERNAME data structure 
220 includes a NAR_USERNAME NAR _ATTR Qualifier 
that provides for Role designation, indicating whether the 
object referenced is acting as a source, destination, both or 
undeterminable within the system. The NAR_A1TK Quali- 
fier bits are set when the Role can be determined without 
ambiguity. 

Referring now to FIG. 11B, a NAR_USER_ID data 
structure 230 is the general type for identifying an account- 
able user. The accounting process 14 can use any available 
object type to represent the NAR_USER_ID value 232. 
The NAR_USER_ID value 232 will be a system estab- 
lished STRING type or a user index as generally supplied by 
a database system. The semantics of the NAR_USER_ID 
value 232 are consistent within the accounting process 14, 
and can be consistent outside of the accounting process 14. 

Referring now to FIG. 1C, a NAR_IP„ADDRESS data 
structure 240 is shown and which is the general network 
component identifier for an IP enterprise network. NAR_ 
IP_ADDRESS data structure 240 includes a IP Address 242 
that is usually -unique within the accountable domain, and 
thus can be usable as an accounting process 14 identifier. 
Within the accounting process 14, the occurrence of this 
record implies that the address is unique within the account- 35 
ing realm. NAR_IP _AD DRESS type includes a NAR_ 
IP _ADDRESS NAR _ATTR Qualifier. The NAR__IP_ 
ADDRESS NAR_ATTR Qualifier provides for Role 
designation, indicating whether the object referenced is 
acting as a source, destination, both or undeterminable 40 
within the system. These bits are set when the Role can be 
determined without ambiguity. 

Referring now to FIG. 11D, a NAR_NETWORK_ID 
type data structure 250 is shown. The NAR_NETWORK_ 
ID data structure 250 includes a NETWORKED value 252 45 
is a general type used for identifying a network component 
when a network address is inappropriate. The accounting 
process 14 can use any available object type to represent the 
NAR_NETWORK_ID, but it is assumed that this value 
will be an accounting process 14 established STRING type, 50 
(e.g., a Media Access Control (MAC) address that is pre- 
defined in Network interface cards), object type or a number 
index that cannot be associated with a network address. The 
semantics of the NAR_NETWORK_ID is consistent 
within the accounting process 14, and can be consistent 
outside the accounting process 14. A NAR_NETWORK_ 
ID NAR _^ATTR Qualifier provides for Role designation, 
indicating whether the object referenced is acting as a 
source, destination, both or undeterminable within the sys- 
tem. These bits are set when the Role can be determined 
without ambiguity. 

Referring now to FIG. IE, a NAR_FLOW_DESC data 
structure 260 is the general type for reporting on flow based 
network activity. The NAR_FLOW _DESC is a composite 
data structure 260 including a IP Source Address 262, IP 
Destination Address 263, Transport Protocol 264, Type of 
Service 265, Source Port 266 and Destination Port 267 that 
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defining, or specifying, the accountable entity. This level of 
flexibility is required because in network accounting, an 
accountable entity could potentially refer to objects that are 
either physical or logical, singular or members of 
collections, or geographically or topologically constrained, 
such as network numbers or autonomous system numbers. 

A set of accountable entities includes Username and 
Network Object Identifiers. There can be additional descrip- 
tive information available within network activity reports 
and within networking components that could be used to 
further describe accountable entities. These entity attribute 
descriptors can be used in the accounting process 14 to 
provide additional flexibility in how network activity infor- 
mation is reported and tallied. Support for entity descrip- 
tions can include object support for: 
Flow Descriptors 

Flow Protocol Descriptors 
Flow Transport Port Descriptors 
Authentication Descriptors 

NAS Descriptors 
Aggregate Descriptors 
Class Identifiers 
Session Identifiers 
Multi-Session Identifiers 
VLAN Identifiers 
ELAN Identifiers 
Group Identifiers 
Access Identifiers 

Source and Destination Ethernet Addresses 
Ingress and Egress Tunnel Ids 
Ingress and Egress Port Numbers 
ATM Virtual Circuit VPI/VCI 
Calling and Called Station Ids 
Flow Status Descriptors 
Class of Service Identifiers 
Quality of Service Identifiers 
Traffic Path Identifiers 
Accounting Time Interval 
Accountable Network Activity Metrics 
Source and Destination Datagrams 
Source and Destination Octets 
Extended Network Activity Attributes 
Network Flow Control Indications 
Host Flow Control Indications 
Traffic Burst Descriptors 
Referring now to FIG. 12, a NET_METRIC data struc- 
ture 270 is shown. A NET_METRIC data structure 270 to 
support a count is shown in FIG. 14. The NET_METRIC 
data structure 270 is used to hold basic accounting values 
that can be tallied within the accounting process 14. The 
NET_METRIC data structure 270 can support time, octets, 
datagram, counts and cells, circuits, tunnels and so forth. 

Referring now to FIGS. 13A and 13B, two basic trans- 
parent objects TRANS _ATTR objects are shown; UNDE- 
FINED 280 and RADIUS 290. New TRANS^ATTR object 
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types can be defined as needed. These are objects that a user Referring now to FIG. 15, a data collection process 330 

may want to send through the accounting process 14, that are performed by the flow data collector 52 of FIG. 17 is shown, 

customer specific, or proprietary in nature. The accounting The flow data collector receives 332 data from the equip - 

process 14 allows for object transparency, i.e., an object that mcnt interface for an network device. The flow data collec- 
thc system does not act on or modify. Thus, the contents of s io r performs an equipment interface specific translation to 

transparent attributes are undefined with respect to the convcrt 336 mc rccc i vcd data mt0 NAR format ^ wcll as 

accounting system. They are passed through, unmodified. populates the NAR header. Once the NAR is populated with 

Flow Data CoUector the appropriate data, the flow data collector 52 attempts to 

Referring to FIG. 14, a flow data ( collector system 300 for CQrrelate 338 ^ newl M NAR ^ fc ^ 

supporting the flow data collector ("FDC") 52 (from FIG. 2) , - , . ( , « . / r ,\ , t . , 

is shown^TTie flow data collector system 300 includes a 10 ™ at *' *5 *7 mad n $ \ ^ ^ 

processor 302 coupled to a memory 304. In this N ^ 10 gently stored m the local store 

embodiment, the FDC is a process stored in the memory 304 314 ( from £ H L G - 14 ) to determine if there are multiple 

and executed by the processor 302. The FDC 52 includes ^stances of the same object. Speaficafly, correlation is 

several NAR processing components or processes. These performed by examining the ACCT_ENTITY_I D (from 

processes include a NAR constructor 306 for converting 15 FIGS. LLA-11E). 

data gathered by the equipment interface (EI) 16 (shown in ^ flow <k la collector uses one clock and one time 
dashed lines) from a network device or technology determinator, so all NARs that the flow data collector is 
("network entity")inio NAR format. Recall that each equip- processing or holding are assumed to be in the same time 
ment interface 42a-42g is associated with an flow data domain. Consequently, the flow data collector need not 
collector. Thus, the combination of a equipment interface 20 consider time during correlation. If the flow data collector 52 
and a flow data collector support a particular device or determines that a NAR ACCT_ENTITY_ID (i.e., the col- 
technology and collects data from the particular device or lection of descriptors or objects as described above) in the 
technology using a pre-defined format, schedule and proto- NAR matches that of another NAR that it is currently 
col specific to that device/technology. The NAR processes holding, the FDC 52 can replace an older (stored) NAR with 
further include a correlator 308, an enhancement process 25 the new (i.e., most recently populated) NAR and discard the 
310 and an aggregator 312 for processing the constructed older NAR. For example, the existing or older NAR may be 
NARs as appropriate. The details of these processes will be a start record and the new NAR a stop record that includes 
discussed farther with reference to FIG. 15 below. all the data included in the older NAR, thus superseding the 
Still referring to FIG. 14, the memory includes a local older NAR. Alternatively, if the new NAR is a replica of an 
store 314 and a flow data collector configuration (file) 318. 30 existing NAR, the FDC may decide to discard the new NAR. 
The local store 314 stores data received from the equipment Also, the data collector can determine that the two NARs 
interface 16 and processed NARs. The configuration file 318 should be merged or aggregated. Thus, the correlation 
is provided at startup to configure the flow data collector 52. process may discard the new NAR, replace an older NAR 
The configuration file 318 specifies various configuration with the new NAR or mark the two matched NARs as 
parameters 319, including a time parameter 320 and a policy 35 candidates for aggregation, a process which is described in 
322. The NAR processes 304 populate and process NARs detail below. 

for data received from network devices via the equipment As part of the correlation process, the flow data collector 

interface 16 in accordance with the policy 322 of the may enhance 340 the new NAR. That is, the FDC may 

configuration file. NARs being held in the local store 314 are determine that the NAR cannot be correlated without some 

transferred to the flow aggregation process 60 (FIG. 2, 40 amount of enhance ment. The FDC 52 enhances the NAR by 

shown here in dashed lines) when the time specified by the supplementing the information provided by the original 

time parameter 320 expires. source equipment with information that is not available from 

It can be appreciated from the above description that the that source equipment. The supplemental information is 
flow data collector 52 is a software component of the added to the ACCT__ENTITY_ID. Recall that the account- 
accounting process and runs on the flow data collector 45 ing entity identifier ACCT_EN T1TY_ID is a collection of 
system 300. The flow data collector system may be any descriptors, so the enhancement process 310 adds to that 
computer system, such as a workstation or host computer, collection of descriptors. For example, the accounting entity 
which can communicate with the equipment interface. ID ACCT_ENTITY_ID in one NAR might include a 
Alternatively, the FDC may reside in the network device source address and a destination address, along with a value 
itself. Many known details of the flow data collector system 50 indicating how long the flow (for the accounting entity) has 
300 have been omitted from FIG. 17 for the sake of clarity, been in existence. A subsequently processed NAR record 
as the figure is intended to highlight the processes of and having those same three objects can be correlated. However, 
memory structures associated with the flow data collector. if a subsequently processed NAR only has two of the three 

Conceptually, as earlier described, each flow data collec- objects, the flow data collector can enhance the accounting 

tor of the accounting process architecture is capable of 55 entity ID ACCT_ENTiTY_JD for the third (missing) 

supporting multiple equipment interfaces 16. At the imple- object to permit correlation. Enhancement may involve 

mentation level, there is a one-to-one correspondence collecting information from a completely different network 

between each flow data collector "process" and a given device (via a NAR generated by another accounting process 

equipment interface 16. For example, a single computer component, such as another data collector), or it may be as 

system might provide both RADIUS and flow probe support 60 simple as adding a timestamp to a NAR's accounting entity 

and thus run separate flow data collector processes for the ID. 

RADIUS EI and the flow probe equipment interface. In such As indicated above, the correlation process may deter- 

a configuration, where the flow data collector processes are mine 342 that two NARs should be "aggregated". Aggre- 

operating independently and loading directly into the flow gation merges the accounting entity identifiers of the two 

aggregation processor 60 (FIG. 2), the computer system 65 NARs together. It also merges metrics for NARs that contain 

itself may be viewed as an flow data collector supporting metrics, as later described. Aggregation of the accounting 

multiple Els. entity identifiers is accomplished through an explicit and 
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implicit matching of those accounting entity identifiers. 
Correlation relies on the explicitly matched fields, that is, the 
fields or objects actually used to determine that two NARs 
should be aggregated. The other descriptors or objects in the 
accounting entity ID that were not used by the correlation 
process to make a match may be equal or different. Aggre- 
gation of the accountable entity ID portion of the NAR keeps 
the explicitly matched objects, and determines which of the 
implicitly matched objects (the matching objects that were 
not a part of the explicit match) to save or discard. Of course, 
the nonmatching objects are automatically discarded, as all 
of the metrics that are the result of this aggregation have to 
apply to the objects in the aggregated accountable entity ID 
ACCT_ENTTTY_ID. The removal of accounting entity ID 
descriptors actually serves to lower the semantic complexity 
of the NAR, whereas enhancement does just the opposite. 

When the data collection process 330 involves a decision 
concerning aggregation, the flow data collector 52 applies 
344 the aggregation policy 322 (from FIG. 14) and uses a 
method defined therein. The method outlines the decision- 
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The flow aggregation processor (FAP) 60 (FIG. 2) aggre- 
gates and/or enhances record data across the system 10. It 
receives data from multiple flow data collectors (FDCs) that 
may be aggregating and enhancing close to the source of the 
information (as described above with reference to FIG. 17). 
As NARs are received from multiple FDCs, the data can be 
further enhanced and/or reduced (i.e. aggregated) to meet 
the specific needs of an application or output interface based 
on the aggregation policy of the flow data processor 60 
(FAP). The design and operation of the FAP will be 
described in more detail below. 
Flow Aggregation Processor 

Referring now to FIG. 16, one implementation of the FAP 
60 is as a database management system, or more specifically, 
a Structured Query Language (SQL) database management 
system, like those commercially available from Oracle or 
Sybase. Although not shown, it will be appreciated that the 
FAP is installed on a computer system, such as a host 
computer. Implemented as a database management system, 



making process to be followed by the FDC in the case of 20 the FAP includes a database server 400 coupled to a database 



implicitly matched objects. The aggregation policy will be 
discussed in further detafl with reference to FIG. 18. Once 
the flow data collector aggregates the accounting entity ID 
ACCT„ENnTY_JD portion of the NAR attributes, it can 
aggregate the NAR metrics. To aggregate the metrics, the 
flow data collector performs a summation process on 
numerical metric values and/or a logical operation (e.g, 
ANDing, ORing, or XORing) on logical metric values. 
Aggregation of the metrics is specific to each metric field in 
the NAR. 

Once the NAR aggregation is complete 346, the FDC 
changes the NAR header (i.e., the NAR„SRC_ID and 
NAR-_SRC_:TIME in the NAR_JD) of the newly aggre- 
gated NAR to identify the component (in this case, the FDC) 
that performed the aggregation as the originator of this 
particular NAR. The FDC stores aggregated NARs for a 
period of time determined by the configuration profile's 
event-based counter or timer 320 (from FIG. 14). When the 
timer expires 348, the FDC is ready to transfer NARs 
processed by the correlator/(enhancement) and possibly the 
aggregator as well to the FAR 

Prior to commencing transfer, the flow data collector 52 
determines 350 if the flow aggregation processor 60 is 
available to receive NARs. If the flow aggregation processor 
60 is unavailable, the flow data collector stores 352 the 
NARs to be transferred in its local store 314 (FIG. 16). The 
flow data collector 52 continues to check 354 the availability 
of the flow aggregation processor at periodic intervals until 
the connection between the flow aggregation processor 60 
and the flow data collector is re-established. When the 
periodic status check indicates 350 that the flow aggregation 
processor is available, the flow data collector loads 356 
NARs into the flow aggregation processor 60. The loading 
function can be implemented according to one of many 
strategies, e.g., a database, file, or data streaming strategy. 
Other strategies could be used. When the flow data collector 
receives 358 a confirmation or acknowledgment back from 
the flow aggregation processor that the NARs were loaded, 
the transfer is deemed successful and the locally stored 
copies of the transferred NARs are removed 360 from the 
local store. Thus, the "store and forward" capabilities of the 
flow data collector provide a measure of fault tolerance at 
this accounting process level to ensure reliable data transfer. 
The flow data collector only transfers NARs when it has 
determined that the flow aggregation processor is available 
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402. The FDCs 52 (from FIG. 14) can use the "push" model 
to move NARs up to the FAP via SQL calls. The database 
402 stores a plurality of tables 404, including a NAR table 
406 (implemented as a persistent cache) and an aggregation 
store 408. Also stored in the database are a plurality of SQL 
commands and procedures (functions) 410 to be executed by 
the server 400. The functions include a FAP correlator 412, 
a FAP enhancer (enhancement process) 414 and a FAP 
aggregator 416. The database also stores a configuration file 
420 for storing configuration parameters such as time and 
policy information. The operation of the FAP will be 
described below with reference to FIG. 17. 

Referring to FIG. 17, an overall flow aggregation process 
430 performed by the FAP is shown. The FAP receives 432 
a NAR from one or more FDCs and loads 434 the received 
NAR into a persistent store or cache (of database 492 from 
FIG. 16), If the FAP is unable to load the NAR, it requests 
436 that the transferring FDC resend the NAR. If the load is 
successful, the FAP sends 438 an acknowledgment back to 
the sending FDC. The FAP determines 440 if the NAR can 
be correlated (with or without enhancement). If the FAP 
determines that the NAR can be correlated, the FAP corre- 
lates 442 the NAR with other NARs received from other 
FDCs. Once the NAR is correlated, it may be enhanced 444 
"across the system", in a manner more fully described with 
reference to FIG. 18. The NAR may be enhanced 446 to 
include enhancement information obtained from an outside 
source (i.e., collected by a data collector for a different 
equipment interface). Once any potential correlation and 
enhancement has been performed, the FAP determines 448 
if the NAR is a candidate for aggregation. If so, the FAP 
applies 450 the aggregation policy 420 (from FIG. 16) and 
stores 452 the resulting aggregated NAR in the aggregation 
store until a predetermined time expires or event occurs 454 
(as set in the FAP configuration 420). The FAP ensures 456 
the uniqueness and integrity of any NAR by examining 
NAR header information prior to re-loading 458 such NAR 
into the persistent store. 

The accounting architecture may be implemented to 
include a second "shadow" FAP process, also coupled to the 
data collectors and operating in the manner described above 
with respect to receiving and processing NARS. In the 
dual/shadowing FAP implementation, the accounting archi- 
tecture further includes an error detection module (not 



and it considers the NAR transfer successful only upon 65 shown) coupled to both of the first (primary) and second 



receipt of an acknowledgment from the flow aggregation 
processor. 



(shadow) FAP processes. The error detection module oper- 
ates to detect an error relating to the first flow aggregation 
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process and cause the aggregate reports from the second resulting, enhanced NAR2 532 has an enhanced accounting 

flow aggregation process to be transferred to the accounting entity ID 534 that includes the T3-to*T4 timestamp (not 

module (i.e., flow data distributor 70) in place of the shown), the IP-to-IP addresses 526-528 and the username 

aggregate reports from the first flow aggregation process. 524. Thus, the enhanced NAR2 now has a mapping between 

Enhancement 5 the username and the one of the IP addresses 526, 528 that 

Now referring to FIG. 18, an example of an application of is related to the IP address 522. The metric 530 is 

the FAP enhancement process 444 (from FIG. 20) is shown. unchanged. 

In the illustrated example, enhancement detcrministically It should be noted that the correlator is able to determine 

identifies the source of a captured network accounting that the time intervals are related to each other because the 

record, flow or a transaction across a network. Enhancement 10 flow data collectors are time synchronized (or closely 

accesses other sources of information on the network in synchronized, assuming some amount of drift). Thus, if the 

order to enhance a record and make it chargeable to a correlator assumes no drift, then T3-to-T4 must be within 

specific user. the time period of Tl-to-T2. The IP-to-username address 

In the example shown in the figure, two NARs of different mapping is an event that has to encompass or cover all of the 

sources are inevitably going to be aggregated together to 15 accounting records that apply to that IP address. Any user 

produce a third unique NAR. A first source equipment (or assigned to this IP address, started at Tl and ended at T2. 

source) 500 is a DHCP (Dynamic Host Configuration Only those records that reference that IP address between Tl 

Protocol) server. A second source equipment (or source) 502 and T2 will have this username applied to it. When the two 

is a flow probe (discussed below). The sources 500, 502 flow data collectors are not strictly synchronized, then the 

have corresponding flow data collectors, a first FDC 20 amount by which T3-to-T4 overlaps Tl-to-T2 should cor- 

(FDC1), 504 and a second FDC (FDC2) 506, respectively, respond to the amount of tolerance, i.e., drift, built into the 

for converting their data into respective NARs NAR1 508 system. The accounting process assumes a drift amount of at 

and NAR2 510. As described earlier, each flow data collec- least one second for even a strict time synchronization, so T4 

tor assigns an accounting entity identifier 512, 514, and adds can be greater than T2 by one second, 

time stamp information 516, 518 on the records of the 25 Referring now to FIG. 19, an aggregation of the enhanced 

sources to which they correspond. The NAR1 508 includes NAR2 532 (from FIG. 18) is shown. In this example, the 

in its assigned accounting entity identifier 512 an "IP aggregation involves combining NARs with IP-to-username 

address-to-username" assignment, thus including an IP address mappings to workgroups. To accomplish this 

address 522 and a username 524. The accounting entity requires two enhancements, two correlations, and an aggre- 

identifier 514 for the second source is an IP-to-IP flow and 30 gation phase. As already described above, with reference to 

therefore includes a first IP address 526 and a second IP FIG. 19, the IP address-to-usemame information is received 

address 528. The NAR2.of the flow probe includes a metric by the FAP and is either passed to the correlation or stored 

530 attribute as well. in the local store but available to the correlator. When the 

These two records NAR1, NAR2 are combined through IP-to-IP address NAR with metrics is received, the correla- 

correlation 442 (from FIG. 17) and enhancement 444 (FIG. 35 tor and the enhancer work together to add the username 

17) to generate an enhanced NAR2 532. This enhanced mappings to these IP-to-IP address NAR. The username 

NAR has a modified accountable entity identifier 534 and a could be provided for one or both of the source and the 

metric. The modified accountable entity identifier is the destination addresses. More than likely, the username is 

existing accounting entity ID 514, to which the FAP has assigned to the source IP address, 

added the IP-to-username assignment 512 from the account- 40 Referring again to FIG. 19, another correlation and 

ing entity ID 512 of the NAR1 508. enhancement process 442, 444 maps the username 524 to a 

Still referring to FIG. 18, the NAR1 508 has an IP-to- workgroup. The FAP builds up search keys using database 

usemame mapping 512 and an accounting interval 516 principles and relational algebra. Thus, for example, the IP 

comprising a start time and a session time to indicate a time address has a one-to-one mapping with a username. (The 

interval bounded by start time "Tl" and a start time +session 45 one-to-one mapping is assured because of the nature of IP 

time ("T2"), that is, the accounting interval represents a start addressing and the way that the DHCP server assigns 

time and a stop time. The username 524 in the IP address- usern antes.) Therefore, there can be only one user for an IP 

to-username mapping is supplied by the DHCP server 500. address in a given instance. These terms or values are 

In the FAP, this NAR1 information will either go directly to equivalent keys, so the username can easily be replaced with 

a correlation function or to the local store (which could 50 the IP address. The username 524 that was inserted into the 

either be a database, file or memory), where it can be directly enhanced NAR2 532 can be used as a look-up into a 

accessed by the correlator function. The NAR2 510 has an workgroup 540 in one of the database tables 404 (FIG. 16) 

accounting entity ID 514, a T3-to-T4 accounting lime inter- because the user is actually a member of a workgroup, 

val 518 and a metric 530. The accounting entity identifier Therefore, the enhancement function can be used to insert 

514 has two IP addresses 526, 528, one corresponding to a 55 the workgroup label into the enhanced NAR2 (already 

source IP address and the other corresponding to a destina- enhanced for username) to produce a twice-enhanced NAR2 

tion IP address. The NAR2 502 is passed to the correlator 542. If the now twice-enhanced record 542 is to be 

442, which determines that the Tl-to-T2 time interval 516 aggregated, it is held in the aggregation store 408 (FIG. 16) 

from the IP-to-username address map in the NAR1 508 for some time period T until other NARs are received for 

overlaps or in some way relates to the T3-to-T4 lime interval 60 potential aggregation. 

518 of the NAR2 510. The correlator determines that T1,T2, Suppose now that another NAR is loaded into the FAP. 

T3 and T4 are related, and that the IP address 522 in the This new NAR passes to correlation, which determines that 

IP-to-username address mapping 512 is associated with one enhancement is need in order to correlate the new NAR with 

of the two IP addresses 526, 528 in the NAR2 510. Thus, the the twice enhanced NAR2 542 of FIG. 19. As a result, the 

FAP enhances the NAR2 510 by inserting information from 65 FAP enhances the NAR to include the username 524 and the 

the accounting entity ID 512 (of NAR1 508) into the workgroup 540 to produce a resultant NAR "NAR3" 550, as 

accounting entity ID portion of the NAR2 510. The shown. 
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Referring to FIG. 20, in addition to the username and the object's accounting entity ID into policy information 

workgroup, the other NAR3 attributes include the T3-T4 572a-g, which includes a collection of data 574 that can be 

accounting interval, the IP-IP address mapping and the supplied by the available flow data collectors and a set of 

metrics. With the enhancement, the correlation process 444 functions or methods 574 needed to correlate, aggregate or 

determines that the resultant NAR3 now matches the twice $ enhance that data in order to construct the accounting entity 

enhanced NAR2 542 held in the aggregation store 408. identifier. 

Having explicitly matched the two NARs, aggregation 448 Aggregation adjustment takes an accounting policy that is 

is performed. Aggregation preserves the explicitly matched a collccticm of accounling objccls and decomposes those 

data objects that are in the accounting entity identifier, accounting objecls iQt0 their account ing entity identifiers 

discards any mismatches in the accounting entity identifier _ . z j *u ?• *j 

. , j - ■ • .* . • . • 10 and then further decomposes the accounting entity ldentin- 
and makes a decision whether to keep the imphcitly matched * i_- * j ^ i, n_ - 
objects (i.e., those that seem to be equal but were not used V s m a f?™™ fasmo ° <° P r0Vlde * c "^tion of basic 
to make the correlation match). It also then combines the d f ta f ^ ( r tlons needed to construct, those accounting 
relevant metric values together via summation or logical identifiers. This concept builds on the logical directed graphs 
operations (e.g., ORing, XORing, ANDing). Once the aggre- as Mn m manv compilers or data flow systems. Knowing 
gation is complete, the FAP holds the resulting aggregated 15 tne order of & G functions, the data requirements and 
NAR 552. As the FAP receives additional NARs, the aggre- dependencies, the data flow software can build the logical 
gator continues to sum and perform these logical operations graph from the decomposition and that specifies data 
on these metrics values for some aggregation period. The requirements and methods that can be distributed to con- 
duration of that aggregation period may be in the order of 60 figuration files in the flow data collectors and FAPs to result 
seconds to a week, or however long the FAP is configured 20 in adjusting the configuration of those accounting compo- 
to aggregate these records. The termination of that period nents. 

can be a time -based or event-based. Once an event that For example, suppose a user wants to receive accounting 

terminates the time period occurs or an aggregation timer on an hourly basis from all of the potential sources of 

expires, the aggregated NARs held in the aggregation store information. The flow data collectors S62a-S62e are the 

are released for output by the FAP. 25 components that are available for collecting the raw infor- 

Aggregation Adjustment mation to generate the accounting data in accordance with a 

It can be understood from the foregoing description that user-specified accounting policy. The internal FAP processes 

aggregation exists at different levels of the accounting 564a-564/> further correlate, enhance and aggregate to 

process. As shown and described above with reference to evolve the data towards the overall accounting data to meet 

FIGS. 15 and 17, both the flow data collector and the FAP 30 the accounting policy 568 specified by a user. Thus, the 

are aggregation-capable. Each aggregates in accordance user's information requirements are translated into a policy 

with an overall aggregation policy that defines how aggre- (i.e., collection of _objects), which is received by the 

gation is used to provide the data to meet the needs of a accounting system and decomposed into the sets of data 

specific application. The aggregation performed by the dif- requirements and methods for each of the available account - 

ferent levels can also be remotely and independently 35 ing components S62a-S62e ) 564a-564£, that is, policy 

adjusted, as will now be described. information 572a-572g). Assuming that these components 

Aggregation adjustment involves the ability to adjust the or processes are already configured, these sets represent 

level of aggregation to meet specific application data needs. configuration updates that are distributed to and stored in the 

There are two aspects of aggregation adjustment: remote configuration files (see FAP configuration file 420 from FIG. 

control and variable degree. 40 16 and FDC configuration file 318 from FIG. 14) in their 

Referring to FIG. 21, a graphical representation of aggre- respective processes, 

gation control and adjustment via a data flow decomposition Referring now to FIG. 22, a depiction of the configuration 

model is depicted. As shown, the accounting system is update is shown. The decomposition/configuration update 

depicted as a tree 560. The flow data collectors are leaf process is implemented in software and is based on known 

nodes 562a-562e and the two illustrated FAP processes are 45 data flow technology used in conjunction with an available 

intermediate nodes 564a-564fe. The root 566 is the collec- visualization tool to act as a front -end graphical interface, 

tive view of all of the processed accounting information. Using such visualization tools, the updated configuration is 

Given a common view of all the data and the particular simply mapped to the appropriate component, 

accounting information requirements of a given application, It should be noted that not all accounting processes have 

the root 566 thus embodies a single accounting/aggregation 50 a complete collection of data collectors. For instance, if the 

policy 568. The accounting policy is defined such that an accounting process is to perform user-based accounting and 

accounting schema is a direct derivative of the accounting the accounting process only has a flow probe, then it will be 

policy 568. necessary to request that the user supply a static table of 

The accounting policy 568 is viewed as a collection of IP-to-username mappings or a source of DHCP user IP 

accounting objects 570, each defined as an accounting entity 55 address mappings. The source of that "outside*' information 

identifier 572 and a set of metrics (not shown). The account- becomes part of the decomposition strategy, 

ing entity identifier is an abstract object resulting from Information Management 

construction functions that use the flow data collector data as The NAR sequence number (NAR_SEQ_NUM FIG. 

its original starting point. If an accounting entity ID is in the 8B) allows components that are in the next level to detect if 

accounting policy as a part of a collection of accounting 60 there are missing NARs in a collection of NARs and can be 

objects, it is there because it can be constructed from the used to give a sense of how often NARs are produced in a 

FDC data and the collective set of operations that allow for given time period. With the time stamps and the sequence 

correlation, enhancement and aggregation. Therefore, if an numbers, a per second creation rate of NARs throughout the 

accounting entity ID can be constructed, it can be decom- system can be determined. With this information being part 

posed. 65 of every NAR, the accounting process 14 can determine a 

To implement a given user/application requirement, sense of the functional capabilities of the intermediate 

therefore, the data flow model 566 decomposes each components and detect some aspects of the communication 
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channel between components. Also included in a NAR 
identifier is a component type identifier 207a which specifies 
what kind of component produced the NAR and its serial 
number 2076 as described above in FIG. 8B. The component 
type identifier allows the accounting process 14 to keep 
component statistics and characteristics based on component 
type. It also allows specific processing on the NARs. NAR 
IDs are allocated in a very specific way through a manage- 
ment system in order to insure that the IDs are actually 
unique within the accounting process 14. 

Referring now to FIG. 23, the sequence numbers (NAR_ 
SEQ_NUM) are a key reliability feature 590 of the account- 
ing process 14. By having the sequence numbers as part of 
the NARS and knowing that the numbers are monotonically 
increasing enables the accounting process 14 to track and 
identify 592 lost traffic or records. It also enables the 
accounting process to determine 592 the amount of lost 
traffic. By having the NARs with stored accounting process 
component IDs (e.g. a data collector assigned to a particular 
network device that is allocated at the time that the collector 
is assigned) the information management process 590, can 
identify 594 the data collector responsible for the flow. The 
accounting process 14 can call back to the data collector that 
produced the NARs of a particular flow and request 596 that 
the missing NARs (i.e., those NARs for which there are 
missing sequence numbers) be retransmitted. 
Flow Probe 

As discussed above in reference to FIG. 2, the accounting 
process supports a flow probe e.g., 12c that captures a user's 
network activity for purposes of IP accounting. The flow 
probe 12c monitors all traffic over a given network link and 
captures data associated with the different flows in the traffic 
on that link. It is capable of monitoring IP data flows over 
a number of technologies (e.g., Ethernet, ATM, FDDI, etc.). 

One important feature of the flow probe is its ability to 
detect and report on successful and unsuccessful connectiv- 
ity. This capability is useful to billing and chargeback 
applications. For example, a user may try to connect to a 
particular switch or reach a particular network, but is 
rejected. The flow probe 12c can identify that transaction as 
unsuccessful and provides the billing application with infor- 
mation that the billing application can use in determining 
whether or not the user should be charged for that transac- 
tion. The flow-based connectivity model embodied in the 
flow probe is described generally with reference to FIGS. 
23-25, and specifically with reference to FIGS. 27-28. 

Referring to FIG. 24, a representation of a network 600 in 
which an end system "A" 602 is connected to another end 
system "B" 604 is shown. The terminal systems A 602 and 
B 604 communicate with one another over a communication 
path 606. Along that path are multiple intermediate devices 
608 (e.g., routers, switches) to support the communication 
services required for communications between A and B. 
Although the path from A to B is depicted as a single straight 
line, it may be appreciated that the actual physical topology 
of this path most likely is extremely complex. For the 
purpose of understanding the flow probe's connectivity 
model, however, it is not necessary to know how the actual 
network would be configured. 

The physical deployment of the flow probe in a network, 
such as the network 600, is based on two criteria: 
performance, e.g., a 100 Mb probe must be deployed within 
a region of the network that operates at 100 Mb, and 
granularity of the information to be generated. That is, if the 
performance or the quality of service provide by A is of 
particular interest, then the flow probe is located as close to 
A as possible so that the flow probe will see all of the traffic 
that is seen by A 
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The deployment of the flow probe may be in-line or 
out -of- line of the stream of IP packets of interest. Thus, the 
flow probe 12 may be deployed in-line, i.e., integrated into 
either of the components that are actually party to a con- 

s versa tion (like end station A 602, as shown in the figure), one 
of the devices 608 that are actually supporting the commu- 
nication or out-of-line, i.e., packets are copied and delivered 
to a remote position. 

Generally, a flow is defined as any communication 
between communicating entities identified by an IP address, 

1 a protocol and a service port. All IP packets (or datagrams) 
are categorized using the fields present in the packets 
themselves: source/destination IP addresses, the protocol 
indicated in the IP header PROTO field, and, in the case of 
UDP or TCP, by the packet's source and destination port 

15 numbers. 

In a given network segment monitored by the flow probe, 
much of the typical IP traffic includes TCP protocol traffic. 
Because the flow probe is a flow based monitor that is 
actually tracking the TCP as a flow, it is completely aware 

20 of the TCP protocol and that protocol's three-way handshake 
algorithm (state machine). The TCP flow has indicators to 
indicate that a connection is being established or a flow is 
being disconnected. However, these messages are only rel- 
evant to the two communicating parties (e.g., A and B in 

25 FIG. 27). The end system A may request that it be able to 
communicate with B and sends a "TCP SYN" indication. 
Any of the networking devices 608 along the path 606 can 
reject this SYN request, completely independent of the 
intended destination (in this example, end system B) and 

30 without the knowledge that the end system B is a party to 
this communication request. There are a variety of problems 
that can cause an internal network component to reject a 
request. For example, a router between A and B may find 
that there is no route available for forwarding a packet 

35 towards B or that the routing path is inoperable (and no 
alternate exits), or the router may find that it doesn't have the 
resources to handle the packet. 

The Internet Control Message Protocol (1CMP) is 
designed to convey this type of error event information back 

40 to the originator of the request. For example, suppose device 
608 is a router that is in a "failed" state and cannot process 
the SYN request that it received from A. The support exists 
in the Internet protocol, specifically, ICMP, to signal this 
condition back to A. Originator A has the ability to correlate 

45 the error event with the request and inform the requesting 
application that its request is not going to be supported. 
Because the network uses a completely independent 
protocol, i.e. ICMP, to convey the information, it is neces- 
sary to correlate these independent protocols (TCP and 

so ICMP) to provide the accounting process with the informa- 
tion it needs to know about a given transaction. Specifically, 
the accounting process needs to know if the transaction was 
successful or unsuccessful and the cause of failure if unsuc- 
cessful. 

55 As an independent monitor operating outside of the 
context of the originating entity ("A", in this example), the 
flow probe is able to produce a complete and accurate record 
of the transaction by mapping the network control informa- 
tion to the user request information. To do so, flow probe 

60 correlates the state information in protocols such as TCP 
with error event or condition messages provided by other 
protocols, such as ICMP. In this manner, it is possible to 
determine if a particular request for a service has actually 
been denied as a result of some network independent event. 

65 The flow probe correlates the dissimilar protocols together 
and finds a way of representing the network event in its 
normal reporting of the TCP flow. 
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The flow probe has specific reporting mechanisms for the 
specific protocols. The TCP protocol, for instance, has many 
more metrics associated with its protocol states than UDP 
based flows. However, because ICMP relevant events or 
network relevant events are not associated with or have any 5 
impact on the state of TCP or UDP or any of the normal 
protocols, the flow probe provides a mechanism for tagging 
its state tracking with the error event. The NAR is repre- 
sented as a start flow indication, a continuing or status record 
and a stop record. All of the flow probe's internal protocol 10 
indications map to start, continuous or stop states. When a 
network rejection event comes in (e.g., in the form of an 
ICMP message, or other type of internet control 
information), regardless of what state the probe is tracking 
as the current state, it reverts to a stop state and has to expand 15 
upon the normal time or transition based stop conditions to 
include an specific ICMP event as the cause of the closed 
state. The flow probe NAR includes bit indications for the 
actual protocol states that it is tracking. For ICMP generated 
events, the flow probe indicates whether the source or the 20 
destination was affected by the events. In order to convey 
this network rejection or network event back to the parent 
flow, the NAR allows for specific network rejection logic to 
be reported either by the source or the destination, and has 
specific bit indicators in either the source or the destination 25 
fields. 

There are two key aspects to the connectivity scheme of 
the flow probe as described thus far. First, the probe deter- 
mines that an ICMP event has occurred. Second, the probe 
correlates that event to the "parent" flow, i.e., the same flow 30 
as that associated with the failed request, and stores the exact 
ICMP event into some state associated with that flow so the 
event can be reported to the accounting system in a NAR. At 
this point it may be useful to examine the IP packet and 
ICMP message formats in general, as well as examine 35 
certain fields of interest. 

Referring to FIG. 25, an exemplary IP packet format 610 
is shown. The IP packet format 610 includes an IP packet 
header 612 and an IP packet data field 614. The IP packet 
header 612 includes a PROTOCOL field 616 for indicating 40 
the protocol of the message encapsulated therein. The header 
also includes a source IP address field 618, a destination IP 
address field 620 and other known fields (not shown). In the 
example of FIG. 25, the message contained in the IP packet 
data field (or payload) is an ICMP message or packet 622. 45 
The ICMP packet is formatted to include an ICMP header 
624 and an ICMP data field 626. In the example, the protocol 
field 616 would be set to indicate a protocol value corre- 
sponding to ICMP. 

Referring to FIG. 26, an exemplary ICMP message format 50 
622 for reporting errors is shown. The format includes an 
ICMP message header 624. The header 624 includes a type 
field 630, which defines the meaning of the message as well 
as its format, and a code field 632 that further defines the 
message (error event). The error reporting message types 55 
(type values) include: destination unreachable (3); source 
quench (4); source route failed (5); network unreachable for 
type of service (11); and parameters problem (12). Each of 
the types has a number of code values. For a destination 
unreachable message (TYPE field value is 3), the possible 60 
codes (code values) include: network unreachable (0); host 
unreachable (1); protocol unreachable (2); port unreachable 
(3); fragmentation needed and DF set (4); source route failed 
(5); destination network unknown (6); destination host 
unknown (7); source host isolated (8); communication with 65 
destination network administratively prohibited (9); com- 
munication with destination host administratively prohibited 



(10); network unreachable for type of service (11); and host 
unreachable for type of service (12). 

Also included in the ICMP message format is a datagram 
prefix field 634, which contains a prefix — header and first 64 
bits of data— of the IP datagram that was dropped, that is, the 
datagram that triggered the error event message. The data- 
gram prefix field 634 corresponds to the ICMP message 
(packet) payload. The IP datagram or packet header 612, 
partially illustrate in FIG. 24, is shown here in its entirety. 
Assuming that the IP datagram carries a TCP message, the 
protocol value would correspond to TCP and the portion of 
the IP datagram's data 636 (first 64-bits) would in fact 
correspond to a TCP message header 636, which includes a 
source port field 638, destination port field 640 and a 
sequence number field 642, The source port is the port 
number assigned to the process in originating (source) 
system. The destination port is the port number assigned to 
the process in the destination system. 

It will be understood that TCP is an example protocol. The 
field 636 could correspond to a portion of packet header 
from a packet of another protocol type. Also, the error 
reporting protocol could be a protocol other than ICMP, and 
the amount of header in field 636 could be more or less than 
64 bits, that is, this amount may be adjusted so that the 
appropriate flow information can be obtained from the 
header of the message contained in the discarded IP packet, 
as described below. 

Referring to FIG. 27, a packet processing method ("the 
process") 650 performed by the flow probe is shown. The 
process captures 652 a new IP packet (datagram) and tests 
654 the received packet to determine if it is good (i.e., 
well-formed). The process 650 examines 656 the protocol 
field in the IP packet header to determine if the protocol is 
the ICMP protocol. If the protocol is ICMP and the infor- 
mation type field is set to one of the five error reporting 
messages described above, the process bypasses the IP 
packet and ICMP message headers and processes 658 the 
ICMP message or packet payload (FIG. 26), which corre- 
sponds to a portion of IP packet which that was discarded 
and to which the event message relates. The payload process 
will be described with reference to FIG. 28 below. Once the 
payload processing is complete, the processing of the IP 
packet resumes 659 the processing that would be performed 
if the IP packet had not been detected as containing an ICMP 
message of the error reporting variety as discussed above, as 
will now be described. 

Still referring to FIG. 27, if the protocol is not ICMP 
and/or the information type is not an error report, the IP 
packet is processed as follows. The probe scans 660 the 
header to determine the values of the fields which corre- 
spond to the "flow key", the fields which define "the flow" 
for the probe. Each flow probe can be configured for a 
particular flow key definition. For example, the flow key 
might be the source/destination IP addresses, the source/ 
destination ports and the protocol. The probe determines 662 
if the flow key of the processed packet header matches a 
flow already stored in the flow probe. A local store in the 
flow probe is used to hold flow representations including 
flow key parameters, metrics, state information. The state 
information will include, in addition to the protocol control- 
related states (i.e., TCP "FIN"), error event/state change 
cause and source/destination to which the message is 
addressed. These flow representations are converted into 
NARs for accounting process reporting purposes. 

Still referring to FIG. 27, if the flow probe cannot match 
664 the flow key information to a stored flow, the probe 
constructs (and stores) 666 a new flow and completes 668 
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the process. If the probe finds a match, it updates 670 metrics Any non-TCP traffic is categorized as a connection -less 

for the matching stored flow (or "parent" flow). It also transaction. When configured to generate the most detailed 

updates 672 the flow state of the parent and then completes level of reporting for connectionless traffic, the flow probe 

674 the process. It should be noted that the construction of can rc port the discovery of a new connection-less transac- 

a new flow triggers 676 the generation of a start NAR and s Uon . Uj e existence of a request/response pair within the 

the update of the flow state triggers 678 the generation of an transaction (as exists when the probe has seen a single 

update NAR. The generation of NARs by the flow probe will packel from both me and ^ destination for the 

be discussed later. transaction); the continuation or transaction persistence, and 

Referring to FIG^28 processing of tr* ICMP message SQ forth ^ transaction persistence status is generated with 
payload^e., the embedded IP packet, 658 (from FIG. 27) is ^ ^ f . hasbccn ^hin a configured timer 
shown. The processing of the ICMP message payload pro- . , # . . , ^ 

cessing is recursive in nature. The essential method is the a / ep0rt Jf f generat ^ DfT - . A . f - fiU 

same as used above for an IP packet (FIG. 27), with a few . ^ status re P ort for Don -T CP fraffic indicates if the report 

differences. If the flow probe determines 664 that there is no f m miUal re P ort > a request/status report or a continuation 

stored flow corresponding to the flow of the dropped IP ( or a current transaction) report. 

packet or datagram (indicated by the ICMP message in the 15 In me default mode, the flow probe generates a status 

data prefix field or payload 634 of FIG. 26), the processing re P ort whei * il has seen a request/response "volley" within a 

is complete 680. If a stored flow matching the flow key of transaction and every 15 minutes thereafter, if the transac- 

the dropped datagram is found, the probe updates 672 the tion persists. This offer immediate notification of request/ 

flow state to indicate a "rejected" state for the stored flow. It response traffic and a fair amount of data reduction on 

also updates the flow slate information to indicate whether 20 connection-less transactions. 

it was the stored flow's source or destination that was Thus, the flow probe state tracking includes protocol- 
associated with the ICMP message and the event cause. The specific state information. It provides detailed information 
state change (to rejected state) triggers 682 the generation of 0 n transport specific flow initiation, such as TCP connection 
a stop NAR, as is later described. Once the probe has establishment, as well as flow continuation and termination 
completed the payload processing 658, it resumes 659 the 25 eV ent reporting, 
processing of the original IPpacket (as indicated in FIG. 27). p ro tocol Independent Packet Monitor 

Thus, the payload processing can be viewed as a packet Referring to FIG. 29A, a network 700 includes a monitor 

processing exception, ao 'exception that is invoked when it m ^ ^ a ^ for detecting packet loss . 

is determined that an ICMP error reporting message has momlor 7Q2 wi]1 te particularly described using IP SEC 

been received. The ICMP messsage reports a error event and , . u j t*. •* hm 

iL Tri , t • . j **u ii_ t , t*l *• 30 authentication headers. The monitor 702 uses sequence 

the IP packet associated with that error event. The exception . . • T ™ ^ x. j ™. 

process serves to correlate the flow of the discarded IP numbers that exist in IP SEC authentication headers. The 

packet-uv the ICMP message with -the parent (matching m ° m t° r 70 t 2 <? an b - e uscd 1° dct ^ ct J 0 ? 1 packe^ts in a n t 

stored) flow, thus mapping the ICMP error (state) informa- of protocol that uses sequence numbers in headers of the 

tion to the parent IP flow. packets, etc. The monitor 702 is an independent monitor that 

The flow probe reports on network traffic activity through 35 can be disposed anywhere in the network 700. The monitor 

a flow probe NAR, which reports IP flow traffic activity. The 702 is protocol independent. 

flow probe categorizes network traffic into one of four The network 700 would include a plurality of such 

classes of traffic flow: I) connection oriented (e.g., TCP); ii) independent monitors 702 each disposed at corresponding 

new connectionless; iii) request/response connectionless single points in the network 70. Typically, the monitor 702 

(e.g., UDP, DNS); and iii) connectionless persistent (e.g., 40 can be disposed in-line such as in a network device such as 

NFS, Multicast BackBONE or "MBONE" multicast traffic). a switch, router, access concentrator, and so forth. 

To each of these class it applies connection oriented seman- Alternatively, the monitor can be disposed in an out of line 

tics for a uniform approach to status reporting. That is, flow arrangement in which network packets are copied from the 

probe treats these dissimilar transaction models as if they device and coupled to the out-of line monitor, 
were the same. There is one uniform structure for the status 45 The monitor 702 examines each packet of a network flow 

reports generated for each of the 4 different transactions. that passes through the device associated with the monitor 

Each status report includes transaction start and stop 702. The monitor 702 receives serialized IP packets. The 

information, MAC and IP source and destination addresses, packets can have the format specified by the Network 

the IP options that were seen, the upper layer protocol used, Working Group, by S. Kent, Request for Comments: 2402, 

the transaction source and destination byte and packet 50 November 1998 "IP Authentication Header" as part of the 

counts and upper layer protocol specific information. The "Internet Official Protocol Standards", The Internet Society 

protocol specific information and the criteria for when the (1998). The IP Authentication header includes a Next 

status reports are created, is different for each of the four Header field that identifies the type of the next payload after 

transaction types. the Authentication Header, Payload Length an 8-bit field 

The connection oriented protocol understood by the flow 55 specifies the length of AH, and a reserved 16-bit field. The 

probe is TCP. Flow probe has complete knowledge of the IP Authentication header also includes a Security Parameters 

TCP state machine and thus can generate status reports with Index an arbitrary 32-bit value that, in combination with a 

each state transition seen within any individual TCP. There destination IP address and security protocol, uniquely iden- 

is also a provision for generating time interval based status tifies the Security Association for a datagram and a 

reporting in the TCP connections that the flow probe is 60 Sequence Number. The sequence number is an unsigned 

tracking. The status report indicates which states were seen, 32-bit field containing a monotonically increasing counter 

if any packets were retransmitted, if the source or destination value (sequence number). It is always present in such 

had closed, and if the report had been generated by a time datagrams and is provided form the purpose to enable an 

condition. In a default mode, the flow probe generates a anti-replay service for a specific security authentication, 

cumulative status at the time a TCP closes, or times out. This 65 According to the standard if anti-replay is enabled the 

strategy offers the greatest amount of data reduction on transmitted Sequence Number is not allowed to cycle. Thus, 

transactions. the sender's counter and the receiver's counter are reset by 
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establishing a new security authentication and thus a new 
key prior to the transmission of the 2 32nd packet. The 
datagram also includes Authentication Data, Le., a variable- 
length field that contains the Integrity Check Value (ICV) for 
the datagram. 

Referring now to FIG. 2 9B, a packet loss detector process 
704 that runs in the monitor 702 is shown. The packet loss 
detector process 704 examines 706 header information in the 
packet, to determine if the packet includes an authentication 
header. If the packet does not include an authentication 
header, then the packet loss detector process 704 ignores 24 
the packet and exits to wait for the next packet. If the packet 
includes an authentication header, the packet loss detector 
process 20 tests 708 to determine if the packet loss detector 
process 20 had been tracking the flow that is represented by 
the source and destination IP addresses and the SPID value 
that is in the authentication header. The packet loss detector 
will perform a cache look up to determine if the flow is 
stored in a cache of currently tracked flows. The packet loss 
detector process 20 tests 708 those values to see if the packet 
loss detector process 704 is currently tracking that security 
flow. 

If the packet loss detector process 704 is not tracking that 
security flow, the packet loss detector process 20 will 
establish 710 a flow cache entry for that flow in a cache that 
can be maintained in memory (not shown). The packet loss 
detector process 704 will store the source and destination IP 
address and the SPID value from of the authentication 
header. The flow cache also includes all other authentication 
headers from other security flows that have previously been 
tracked. The flow cache enables the packet loss detector 
process 20 to monitor and track. many hundreds, thousands, 
and so forth of different security flows. A cache entry is 
established for every different flow. Once the cache entry is 
established, the packet loss detector process 704 updates 712 
the sequence number entry in the cache for that security 
flow. That is, the initial sequence number in the authentica- 
tion header for the encountered flow is stored. The sequence 
number can start at any arbitrary value. 

If, however, the packet loss detector process 704 deter- 
imined 708 that it is tracking the flow, then the packet loss 
detector process 704 tests 714 if the sequence number in the 
current packet is equal to the previous sequence number 
noted for this flow plus 1. If the sequence number in the 
current packet is equal to the previous sequence number plus 
1, then the packet loss detector process 704 can stop the 
current evaluation because the packet loss detector process 
704 did not detect and the system did not experience any 
packet loss on that particular association. The packet loss 
detector process 704 will update 712 the stored sequence 
number for that flow in the cache. 

If the sequence number in the current packet does not 
equal the previous sequence number noted for this flow plus 
1, the packet loss detector process 704 for the IP SEC 
Authenication packets detected a potentially missed packet. 

For some protocols that permit wrap around, the packet 
loss detector process 704 tests 718 if the sequence number 
has wrapped around e.g., gone from 32 bits of all ones to 32 
bits of all zeros. The IP SEC Authentication packets cur- 
rently do not permit wrap around, so test 718 would not be 
necessary for IP SEC Authentication Headers. If for other 
protocols (or latter versions of the IP SEC Authenication 
protocol), the packet loss detector process 704 detects a 
wrap around condition then there has not been any packet 
loss and the packet is dropped. The packet loss detector 
process 704 will update 712 the stored sequence number for 
that flow in the cache. If the sequence number is any other 



number, i.e., it did not turn over to all zeros, then there may 
have been packet loss. If there may have been packet loss, 
the packet loss detector process 704 can determine how 
many packets have been lost by determining how many 

S sequence numbers are missing. 

When packets may traverse more than one packet monitor 
10, the packet loss detector process 704 may produce a 
packet loss detected indication that docs not indicate that the 
packets were actually dropped. Apacket loss drop indication 

10 in a multi-monitor embodiment indicates that the lost pack- 
ets did not come through the particular packet loss detector 
process 704. However, the indicated lost packets could be on 
other segments of the network. That is, it is possible that 
other parts of the current flow are in other parts of the 

is network. Therefore, the packet loss detector process 704 
notes how many packets were actually successfully 
transmitted, as well as lost, and optionally their sequence 
numbers. These values can be compared to other values 
from other monitors 702 to establish whether or not there 

20 had been packet loss for the flow through the network. 
This indication, could be converted into Network 
Accounting Records thus would be coupled to a process e.g. 
the accounting process 14 that reports statistics on that 
particular flow to provide a summary of how many packets 

25 were lost relative to how many packets were actually 
successfully transmitted on the flow. In the accounting 
process 14, the network accounting records are correlated, 
aggregated, enhanced and so forth to identify network flows. 
This information can be used to determine the records that 

30 correspond to a particular network flow and whether a 
determined network flow lost any packets. 
Capturing Quality of Service 

Referring now to FIG. 30, a process 730 for capturing 
quality of service in a network system 11, (FIG. 1), is shown. 

35 The capturing quality of service process 730 allows an 
administrator to configure 732 the network 11 with a policy 
that corresponds to a first quality of service. The process 730 
also includes an optimization process that assigns or devel- 
ops 734 the policy, defines the policy being used, and 

40 enforces the policy by deploying a policy dictated configu- 
ration into various policy enforcement devices in the net- 
work 11. The capturing quality of service process 730 allows 
the administrator to observe 736 the actual service delivered 
by the network 11 to a customer on the network 11 to 

45 determine if the quality of service provided matches that 
specified by the policy 740. 

The capturing quality of service process 730 uses an 
accounting process 738 to collect information from the 
network. A preferred accounting process is accounting pro- 

50 cess 14 described above. The accounting process 14 collects 
data from the network 11 as part of the observation process 
736. The accounting process collects different kinds of 
metrics from the network, correlates these metrics to speci- 
fied network flows, via the use of NARS, and maps 

55 collected, correlated information i.e., NARs back to the 
policy that was defined and actually deployed in the net- 
work. Because the accounting process 14 performs this 
observation function, the accounting process can provide an 
indication 738a whether or not the policy 740 is being 

60 satisfied. 

By deploying the accounting process 14 to observe ser- 
vice quality, the capturing quality of service process 730 can 
validate performance of service level agreements (not 
shown). If the capturing quality of service process 730 
65 detects that the policy level specified in a service level 
agreement is not being enforced, then the policy can be 
reassessed, redefined, and redeployed 742. The capturing 
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quality of service process 730 can again observe 737. 
Through the observation 736, the capturing quality of ser- 
vice process 730 can determine whether reassessment and 
redefining of the deployed policy was successful. Several 
cycles of this quality of service optimization process could 5 
be required. 

An important component of quality of service includes 
determining whether there has been packet loss. The packet 
detector monitor described in conjunction with FIGS. 29A 
and 29B can be used to access packet loss. The packet 10 
detection monitor 702 can be deployed in the network and 
generate NARs that can be used to determine packet loss as 
discussed above. This information can be used in the cap- 
turing quality of service process 730 to assess whether the 
policy specified by the service level agreement was provided 15 
to the customer. Additionally, so called Differentiated Ser- 
vice "DivServe technology" that a known quality of service 
solution that has been proposed for the Internet as well as 
enterprise networks. In contrast to a per-flow orientation of 
some types of quality of service solutions such as Int-serv 20 
and RSVP, DiffServ enabled networks classify packets into 
one of a small number of aggregated flows or "classes", 
based on bits set in the type of service (TOS) field of each 
packet's IP header. This is a quality of service technology for 
IP networking is designed to lower the statistical probability 25 
of packet loss of specific flows. The capturing quality of 
service process 730 establishes DivServ policy, that is 
decomposed into a collection of DivServ configurations. 
The DivServ configurations are deployed to a collection of 
routers or switches that the customer would have access to 30 
in the network 11 as part of the enforcement/deployment 
process 732. Because packet loss is a statistical 
phenomenon, the capturing quality of service process 730 
observes 736 a large number of network flows. The captur- 
ing quality of service process 730 can observe network 35 
traffic because of the use of the accounting process 14 and 
the resulting NARs at the granularity in which the DivServe 
policies are actually being deployed. The DivServe policies 
are generally deployed at the source and destination IP 
address, protocol and possibly destination port level. 40 

By observing 736 network flows at the same granularity 
as a DivServe policy enforcement mechanism, if the cap- 
turing quality of service process 730 detects packet loss at 
that granularity, then there will be a direct feedback coupling 
to determine whether the policy is actually being enforced or 45 
not. If the policy is not being enforced, then an administrator 
will can reassess the policy, redefine the policy, and redeploy 
742 new enforcement strategies. The capturing quality of 
service process 730 again will observe 736. 

As mentioned, because IP network quality of service is a so 
statistical phenomenon, the capturing quality of service 
process 730 obtains a large number of samples, over a long 
period of time. Through this optimizing capturing quality of 
service process 730 and DivServe deployment 734, the 
customer will get beneficial policy deployment for this 55 
service. 

Service Management 

Referring now to FIG. 31, a service management loop 750 
includes a service provisioning application 752, a policy 
enabled network server 754 and an accounting process 756. 60 
In a typical example, an Internet Service Provider (ISP) and 
a customer will enter into a service agreement or contract 
751 that will specify a level of service for the network. The 
contract 751 has requirements and conditions that are 
enforced by the policy enabled network 754. The service 65 
contract 751 is decomposed by the policy server to produce 
a template that defines the service represented by the agree- 
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ment 751. The template is fed to the service provisioning 
application 752 that actually produces a configuration file 
752a that is sent out to the network 10 to configure network 
for a level of service based upon that contract 751. 

A service management feedback process 750 therefore 
includes three components, service provisioning 752, policy 
server 754 and service accounting 756. The role of service 
provisioning 752 is to send requests 7526 to the policy 
server 754 to obtain an appropriate active policy, and 
obtaining rules and domain information 754a from the 
policy server. The provisioning system can communicate 
with appropriate network management systems and element 
management systems (not shown) to configure the network 
10 for an end-to-end service. When the configuration 752a 
is deployed at the various network devices (not shown) at 
that point, the service is produced. The level of service is 
monitored or audited by the accounting system 756 which 
can be the accounting process 14 described above. The 
accounting process 14 monitors the level of service by 
producing appropiate network accounting records. The net- 
work accounting records NARs are used by a billing appli- 
cation to adjust billing based on the level of service that was 
provided as determined by the accounting system 14. The 
accounting system 14 also can compare the policies pro- 
duced by the policy server to the actual levels of service 
provided to the customer by examining NARs that are 
produced by the customer's usage of the network. 

In addition, levels of service might change, and the 
system takes changes into account so that the service man- 
agement can modify the charge or account differently for 
those changes in levels of service. The service accounting 
also uses the active policy information from the policy 
server to deliver billing information to a billing system or to 
a chargeback system that can may adjustments to billings for 
the service. 

A policy enable network 754 is build on the capabilities 
of address management, domain name management and so 
forth. Essentially in a policy enabled network, policy ser- 
vices produce a set of rules and applys those rules to a 
domain or problem set. The policy server communicates the 
rules to the accounting process 14 so that the accounting 
process 14 can determine what kind of records to generate. 
All of the information is described using data flows. 

As an example, a service contract may specify that a 
company "X" will be given 100% availability of a particular 
network device e.g., a router (not shown) and its correspond- 
ing service. In order to assure that level of service, the policy 
server 754 sends that requirement in a template to the 
provisioning service 752 to produce a configuration file 
752a to configure the router to give company "X" preferred 
use for the router. Therefore, every time a packet from 
company "X's" network comes across the router, the packet 
will always be transmitted unless there is something wrong 
with the router. This may occur even if a packet of company 
«Y" which has a lower service level than company "X" is 
waiting in the router to be transmitted. The packet from 
company "Y" will wait because company "Y" is not paying 
for the quality of service that company X" is paying for. 

In that case, the provisioning service configures 752 the 
policy enforcement mechanism that was put into the router 
in the network. How the policy was defined to the provi- 
sioning equipment is that there is a one-to-one relationship 
between the policy and what the accounting process 14 will 
monitor in the network. The accounting process 14 will be 
aware that company "X" contracted to have 100% avail- 
ability from the router. 

The accounting process 14 will then take every source of 
information it has available and will construct an accounting 
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record that reflects the level of service actually delivered to 
company "X," The accounting records produce are relative 
to the two components, i.e., the router and the customer. The 
accounting process 14 is flexible and can generate account- 
ing records of any flow abstraction. In this process 750, the s 
policy server 754 sends a flow based policy to the provi- 
sioning server 752. The provisioning server 752 uses a flow 
based policy to configure the network. That same flow based 
policy is passed to the accounting process 14 which can 
generate network accounting records NARs having metrics 10 
that can be used to match the same level of those flows. The 
output of the accounting process 14 will determine whether 
the quality of service, availability, etc. that was contracted 
for in the contract 751 was provided. Therefore the service 
management process 750 provides the level of service that 15 
was delivered at the same semantic level as the actual 
contract. 

Capturing quality of service as audited by the accounting 
process 14 includes detecting of packet loss, as mentioned 
above. Each of the components managed by the service 20 
management process 750 require information. Therefore, the 
service provisioning has to provision these various quality 
levels. The policy server 754 thus, keeps what is essentially 
enforcement of the levels of quality that are offered by 
different service types, and the accounting process 756 25 
detects, monitors and audits whether those classes in quality 
of service are being delivered. 

Referring to FIG. 32, an implementation of the service 
management provisioning 752 is shown. The service man- 
agement provisioning 752 extends concepts of device man- 30 
agement and network management into a service manage- 
ment layer of functionality. Service management 
provisioning includes a provisioning core 782, provisioning 
modules 784, and element managers 786. Service provision- 
ing 752 is user focused rather that network focused as 35 
conventional network management. Network management 
involves communication with network systems and equip- 
ment. Service provisioning 752 is orient more towards a user 
and a user's concepts of services. Service provisioning 752 
provides an additional layer of abstraction that relates 40 
description of services at a user level to a network's ability 
to provide those end-to-end services. The architecture 780 of 
Service provisioning 752 is multi-device 788 at the bottom 
of the architecture and multi-service 790 at the top of the 
architecture. The service provisioning 752 is deployed to 45 
write commands to the network systems i.e., network ele- 
ments 788 in order to change configurations of those sys- 
tems. 

Since many end customer services now require that a 
network operate with multiple, different kinds of network 50 
elements in order to provide an end-to-end service, the 
service provisioning 752 simplifies producing information 
that is necessary for a service provider to translate a service 
order from a customer to a network configuration, i.e., all 
commands necessary for all the different elements in the 55 
network in order to create an end-to-end service. 

The service provisioning builds on existing systems. That 
is, in the lower layers there are existing element managers 
that have a configuration management system to configure at 
the network layer. The service provisioning adds layering 60 
over the conventional network management layer. Service 
provisioning maps a customer specified end to end service to 
the network elements that are required to produce that 
end-to-end service. Mapping of a customer's service orders 
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into the state of the network can have various pieces of 
workflow necessary to create or completely activate this 
service order. 

OTHER EMBODIMENTS 

It is to be understood that while the invention has been 
described in conjunction with the detailed description 
thereof, the foregoing description is intended to illustrate 
and not limit the scope of the invention, which is defined by 
the scope of the appended claims. Other aspects, advantages, 
and modifications are within the scope of the following 
claims. 

What is claimed is: 

1. A method for tracking network accounting records in a 
accounting process that collects and correlates information 
derived from network data comprises: 

producing a network accounting record that has an iden- 
tifier that uniquely identifies the record within the 
accounting process with the identifier including a 
sequence number that specifies a sequence number for 
network accounting records that originate from the 
source device; 

determining when there is a break in the sequence num- 
bers of network accounting records produced from the 
source device; and 

requesting missing network accounting records when 
there is a break in the sequence. 

2. The method of claim 1 wherein producing a network 
accounting record further comprises: 

producing a network source identifier that identifies a 
source device that creates the network accounting - 
record. 

3. The method of claim 2 further comprising determining 
the data collector that produced the missing network 
accounting records. 

4. The method of claim 3 wherein determining the data 
collector comprises: 

examining the network source identifier in a data flow. 

5. The method of claim 4 wherein the data flow is 
identified by aggregating received network accounting 
records and correlating the received records to identify a 
flow. 

6. A system comprising: 

a data collector collecting data from a network device, and 
producing network accounting records from the col- 
lected data; and 

a flow aggregation process, that receives network 
accounting records, the network accounting records 
including data identifying the data collector and a 
sequence number, said flow aggregation process 
detects missing network accounting records by detect- 
ing at least one missing sequence number; 

wherein upon detecting a missing sequence number, the 
flow aggregation process retrieves data identifying the 
data collector from received records that have been 
correlated to identify a flow associated with the missing 
records; and 

sends a request to the identified data collector to retrans- 
mit the missing record corresponding to the missing 
sequence number. 

* * * * * 
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